[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:04:50.944291 25352 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.194.62:36531
I20240502 08:04:50.945596 25352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:04:50.946688 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:50.963835 25361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:50.964298 25358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:50.965390 25352 server_base.cc:1034] running on GCE node
W20240502 08:04:50.964684 25359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.069272 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:52.069518 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:52.069706 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637092069682 us; error 0 us; skew 500 ppm
I20240502 08:04:52.070593 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:52.075959 25352 webserver.cc:460] Webserver started at http://127.24.194.62:43291/ using document root <none> and password file <none>
I20240502 08:04:52.076839 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:52.077065 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:52.077471 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:52.083810 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/master-0-root/instance:
uuid: "596753f1d4ec462ea98c4a953b0266ad"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.091558 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240502 08:04:52.096690 25367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.097762 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:04:52.098127 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/master-0-root
uuid: "596753f1d4ec462ea98c4a953b0266ad"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.098465 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:52.163702 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:52.165277 25352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:04:52.165745 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:52.233789 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.62:36531
I20240502 08:04:52.234022 25418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.62:36531 every 8 connection(s)
I20240502 08:04:52.240667 25419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:52.264017 25419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: Bootstrap starting.
I20240502 08:04:52.269361 25419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:52.271257 25419 log.cc:824] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: Log is configured to *not* fsync() on all Append() calls
I20240502 08:04:52.275816 25419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: No bootstrap required, opened a new log
I20240502 08:04:52.298301 25419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER }
I20240502 08:04:52.298851 25419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:52.299049 25419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596753f1d4ec462ea98c4a953b0266ad, State: Initialized, Role: FOLLOWER
I20240502 08:04:52.299830 25419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER }
I20240502 08:04:52.300297 25419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:04:52.300526 25419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:04:52.300817 25419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:52.304774 25419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER }
I20240502 08:04:52.305471 25419 leader_election.cc:302] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 596753f1d4ec462ea98c4a953b0266ad; no voters:
I20240502 08:04:52.307178 25419 leader_election.cc:288] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:04:52.307497 25422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:52.309736 25422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 1 LEADER]: Becoming Leader. State: Replica: 596753f1d4ec462ea98c4a953b0266ad, State: Running, Role: LEADER
I20240502 08:04:52.310725 25422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER }
I20240502 08:04:52.311791 25419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:04:52.320873 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "596753f1d4ec462ea98c4a953b0266ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER } }
I20240502 08:04:52.321684 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:52.321601 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596753f1d4ec462ea98c4a953b0266ad. Latest consensus state: current_term: 1 leader_uuid: "596753f1d4ec462ea98c4a953b0266ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596753f1d4ec462ea98c4a953b0266ad" member_type: VOTER } }
I20240502 08:04:52.322476 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:52.325497 25431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:04:52.337080 25431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:04:52.341934 25352 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:04:52.355938 25431 catalog_manager.cc:1347] Generated new cluster ID: 35a6c1f76aed4934be126ba1e76ba6b5
I20240502 08:04:52.356269 25431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:04:52.371953 25431 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:04:52.373662 25431 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:04:52.394765 25431 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: Generated new TSK 0
I20240502 08:04:52.395687 25431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:04:52.408286 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:52.416887 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:52.419627 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:52.423300 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.424244 25352 server_base.cc:1034] running on GCE node
I20240502 08:04:52.424968 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:52.425171 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:52.425340 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637092425321 us; error 0 us; skew 500 ppm
I20240502 08:04:52.425915 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:52.430613 25352 webserver.cc:460] Webserver started at http://127.24.194.1:38201/ using document root <none> and password file <none>
I20240502 08:04:52.431144 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:52.431331 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:52.431607 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:52.432737 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-0-root/instance:
uuid: "e14b3f66048a41d5bfe92033b1925327"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.438491 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:04:52.442940 25449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.443742 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20240502 08:04:52.444008 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
uuid: "e14b3f66048a41d5bfe92033b1925327"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.444290 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:52.464497 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:52.465798 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:52.468509 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:52.473186 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:52.473415 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.473716 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:52.473888 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.576963 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.1:43785
I20240502 08:04:52.577098 25511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.1:43785 every 8 connection(s)
I20240502 08:04:52.584287 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:52.598623 25516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:52.610582 25517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.611470 25352 server_base.cc:1034] running on GCE node
W20240502 08:04:52.612200 25519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.613236 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:52.613449 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:52.613713 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637092613693 us; error 0 us; skew 500 ppm
I20240502 08:04:52.614650 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:52.619354 25512 heartbeater.cc:347] Connected to a master server at 127.24.194.62:36531
I20240502 08:04:52.619839 25512 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:52.620041 25352 webserver.cc:460] Webserver started at http://127.24.194.2:40797/ using document root <none> and password file <none>
I20240502 08:04:52.620610 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:52.620815 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:52.621093 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:52.620970 25512 heartbeater.cc:511] Master 127.24.194.62:36531 requested a full tablet report, sending...
I20240502 08:04:52.622308 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-1-root/instance:
uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.624475 25384 ts_manager.cc:194] Registered new tserver with Master: e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785)
I20240502 08:04:52.627480 25384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45128
I20240502 08:04:52.628342 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240502 08:04:52.632859 25524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.633723 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:04:52.634002 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.634330 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:52.651038 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:52.652268 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:52.654713 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:52.658139 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:52.658344 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.658568 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:52.658725 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.754680 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.2:39813
I20240502 08:04:52.754781 25586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.2:39813 every 8 connection(s)
I20240502 08:04:52.763662 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:52.772696 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:52.775419 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.776299 25587 heartbeater.cc:347] Connected to a master server at 127.24.194.62:36531
I20240502 08:04:52.776719 25587 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:52.777626 25587 heartbeater.cc:511] Master 127.24.194.62:36531 requested a full tablet report, sending...
I20240502 08:04:52.779839 25384 ts_manager.cc:194] Registered new tserver with Master: c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:52.781515 25384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45134
I20240502 08:04:52.783604 25352 server_base.cc:1034] running on GCE node
W20240502 08:04:52.784757 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:52.785682 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:52.785892 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:52.786082 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637092786063 us; error 0 us; skew 500 ppm
I20240502 08:04:52.786626 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:52.789587 25352 webserver.cc:460] Webserver started at http://127.24.194.3:42607/ using document root <none> and password file <none>
I20240502 08:04:52.790112 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:52.790308 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:52.790552 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:52.791671 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-2-root/instance:
uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.796281 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 08:04:52.799736 25598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.800562 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:04:52.800918 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
format_stamp: "Formatted at 2024-05-02 08:04:52 on dist-test-slave-753s"
I20240502 08:04:52.801276 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714637090785390-25352-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:52.824641 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:52.825840 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:52.827229 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:52.829268 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:52.829464 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.829691 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:52.829862 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:52.918139 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.3:34147
I20240502 08:04:52.918304 25660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.3:34147 every 8 connection(s)
I20240502 08:04:52.935529 25661 heartbeater.cc:347] Connected to a master server at 127.24.194.62:36531
I20240502 08:04:52.935901 25661 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:52.936689 25661 heartbeater.cc:511] Master 127.24.194.62:36531 requested a full tablet report, sending...
I20240502 08:04:52.938542 25384 ts_manager.cc:194] Registered new tserver with Master: f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:52.939152 25352 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016220647s
I20240502 08:04:52.940258 25384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45140
I20240502 08:04:52.942838 25352 test_util.cc:268] Using random seed: 249033690
I20240502 08:04:52.993213 25384 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45144:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:52.997185 25384 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:53.121027 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 1c877e979fbd43fc947a2cc17c3e17cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 08:04:53.122473 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c877e979fbd43fc947a2cc17c3e17cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.126274 25477 tablet_service.cc:1467] Processing CreateTablet for tablet f2a7530ae84444f28528e2ecc8d1ed96 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 08:04:53.127193 25474 tablet_service.cc:1467] Processing CreateTablet for tablet 62009aa71d334e60a3ab5c131817fc2e (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 08:04:53.131139 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a7530ae84444f28528e2ecc8d1ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.126242 25475 tablet_service.cc:1467] Processing CreateTablet for tablet 478eeaef42cf494eb349a9ca2a4d7b17 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 08:04:53.132766 25473 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3937efc374225b60593697f9d6f79 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 08:04:53.132252 25552 tablet_service.cc:1467] Processing CreateTablet for tablet f2a7530ae84444f28528e2ecc8d1ed96 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 08:04:53.144250 25474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62009aa71d334e60a3ab5c131817fc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.144507 25551 tablet_service.cc:1467] Processing CreateTablet for tablet 1c877e979fbd43fc947a2cc17c3e17cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 08:04:53.152433 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478eeaef42cf494eb349a9ca2a4d7b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.154016 25548 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3937efc374225b60593697f9d6f79 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 08:04:53.159674 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a7530ae84444f28528e2ecc8d1ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.156268 25472 tablet_service.cc:1467] Processing CreateTablet for tablet 010bb756381048b98401bce75b15f03b (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 08:04:53.162838 25625 tablet_service.cc:1467] Processing CreateTablet for tablet 1c877e979fbd43fc947a2cc17c3e17cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 08:04:53.155831 25547 tablet_service.cc:1467] Processing CreateTablet for tablet 010bb756381048b98401bce75b15f03b (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 08:04:53.155145 25624 tablet_service.cc:1467] Processing CreateTablet for tablet 478eeaef42cf494eb349a9ca2a4d7b17 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 08:04:53.166935 25626 tablet_service.cc:1467] Processing CreateTablet for tablet f2a7530ae84444f28528e2ecc8d1ed96 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 08:04:53.157680 25550 tablet_service.cc:1467] Processing CreateTablet for tablet 478eeaef42cf494eb349a9ca2a4d7b17 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 08:04:53.157470 25549 tablet_service.cc:1467] Processing CreateTablet for tablet 62009aa71d334e60a3ab5c131817fc2e (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 08:04:53.171809 25473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3937efc374225b60593697f9d6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.165388 25471 tablet_service.cc:1467] Processing CreateTablet for tablet 0933e210c5b442f38d9598dcc8cb87b1 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 08:04:53.173067 25624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478eeaef42cf494eb349a9ca2a4d7b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.173774 25623 tablet_service.cc:1467] Processing CreateTablet for tablet 62009aa71d334e60a3ab5c131817fc2e (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 08:04:53.173121 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c877e979fbd43fc947a2cc17c3e17cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.175855 25546 tablet_service.cc:1467] Processing CreateTablet for tablet 0933e210c5b442f38d9598dcc8cb87b1 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 08:04:53.175161 25470 tablet_service.cc:1467] Processing CreateTablet for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 08:04:53.183862 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a7530ae84444f28528e2ecc8d1ed96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.179428 25620 tablet_service.cc:1467] Processing CreateTablet for tablet 0933e210c5b442f38d9598dcc8cb87b1 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 08:04:53.186379 25545 tablet_service.cc:1467] Processing CreateTablet for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 08:04:53.174585 25622 tablet_service.cc:1467] Processing CreateTablet for tablet d8f3937efc374225b60593697f9d6f79 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 08:04:53.185432 25547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010bb756381048b98401bce75b15f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.192044 25469 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd0117fac064eaea657db7a0263b0a2 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 08:04:53.190945 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010bb756381048b98401bce75b15f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.182653 25621 tablet_service.cc:1467] Processing CreateTablet for tablet 010bb756381048b98401bce75b15f03b (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 08:04:53.197698 25544 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd0117fac064eaea657db7a0263b0a2 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 08:04:53.199487 25545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.202700 25544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd0117fac064eaea657db7a0263b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.199681 25625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c877e979fbd43fc947a2cc17c3e17cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.205642 25471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0933e210c5b442f38d9598dcc8cb87b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.206239 25546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0933e210c5b442f38d9598dcc8cb87b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.209172 25619 tablet_service.cc:1467] Processing CreateTablet for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 08:04:53.211613 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62009aa71d334e60a3ab5c131817fc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.211915 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3937efc374225b60593697f9d6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.215451 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd0117fac064eaea657db7a0263b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.217037 25621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010bb756381048b98401bce75b15f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.216063 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62009aa71d334e60a3ab5c131817fc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.219177 25618 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd0117fac064eaea657db7a0263b0a2 (DEFAULT_TABLE table=test-workload [id=5668772ab8a64a84a2191118044c9586]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 08:04:53.230651 25620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0933e210c5b442f38d9598dcc8cb87b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.230799 25470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.248905 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd0117fac064eaea657db7a0263b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.254896 25619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f86f3a4a7944c498b4b6bbbb5d606cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.273468 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f3937efc374225b60593697f9d6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.279372 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478eeaef42cf494eb349a9ca2a4d7b17. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:04:53.347175 25432 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:04:53.480018 25679 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.556694 25681 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:53.562094 25680 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:53.563459 25679 tablet_bootstrap.cc:654] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.567698 25681 tablet_bootstrap.cc:654] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.571405 25680 tablet_bootstrap.cc:654] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.579531 25679 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:53.580071 25679 ts_tablet_manager.cc:1367] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.011s
I20240502 08:04:53.582020 25681 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.582485 25681 ts_tablet_manager.cc:1367] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.006s
I20240502 08:04:53.582979 25680 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:53.583581 25680 ts_tablet_manager.cc:1367] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20240502 08:04:53.582964 25679 raft_consensus.cc:348] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.583989 25679 raft_consensus.cc:374] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.584304 25679 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.585067 25679 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.586391 25680 raft_consensus.cc:348] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.587172 25680 raft_consensus.cc:374] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.587472 25680 raft_consensus.cc:729] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.588161 25680 consensus_queue.cc:260] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.593502 25681 raft_consensus.cc:348] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.594609 25681 raft_consensus.cc:374] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.595017 25681 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.595937 25681 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.599524 25661 heartbeater.cc:503] Master 127.24.194.62:36531 was elected leader, sending a full tablet report...
I20240502 08:04:53.602414 25681 ts_tablet_manager.cc:1398] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.020s user 0.007s sys 0.002s
I20240502 08:04:53.603816 25681 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:53.613025 25679 ts_tablet_manager.cc:1398] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.033s user 0.021s sys 0.011s
I20240502 08:04:53.613943 25512 heartbeater.cc:503] Master 127.24.194.62:36531 was elected leader, sending a full tablet report...
I20240502 08:04:53.614410 25679 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.622236 25679 tablet_bootstrap.cc:654] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.629107 25587 heartbeater.cc:503] Master 127.24.194.62:36531 was elected leader, sending a full tablet report...
I20240502 08:04:53.630903 25681 tablet_bootstrap.cc:654] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.636862 25680 ts_tablet_manager.cc:1398] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.053s user 0.012s sys 0.002s
I20240502 08:04:53.639489 25680 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:53.640669 25679 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:53.641139 25679 ts_tablet_manager.cc:1367] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.015s
I20240502 08:04:53.643499 25686 raft_consensus.cc:482] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.647063 25681 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.646889 25686 raft_consensus.cc:504] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.647539 25681 ts_tablet_manager.cc:1367] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.002s
I20240502 08:04:53.650296 25681 raft_consensus.cc:348] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.651031 25681 raft_consensus.cc:374] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.644048 25679 raft_consensus.cc:348] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.651331 25681 raft_consensus.cc:729] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.651502 25679 raft_consensus.cc:374] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.651876 25679 raft_consensus.cc:729] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.652070 25681 consensus_queue.cc:260] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.652539 25679 consensus_queue.cc:260] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.654808 25679 ts_tablet_manager.cc:1398] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240502 08:04:53.655774 25679 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.657960 25681 ts_tablet_manager.cc:1398] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 08:04:53.658864 25681 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:53.645501 25680 tablet_bootstrap.cc:654] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.669224 25686 leader_election.cc:288] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:53.675462 25679 tablet_bootstrap.cc:654] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
W20240502 08:04:53.676445 25662 tablet.cc:2369] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:04:53.685091 25681 tablet_bootstrap.cc:654] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.685510 25680 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:53.686009 25680 ts_tablet_manager.cc:1367] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.005s
I20240502 08:04:53.688899 25680 raft_consensus.cc:348] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.689656 25680 raft_consensus.cc:374] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.689960 25680 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.690721 25680 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.693183 25680 ts_tablet_manager.cc:1398] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240502 08:04:53.694872 25680 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:53.700750 25680 tablet_bootstrap.cc:654] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.704551 25679 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:53.704813 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:53.705085 25679 ts_tablet_manager.cc:1367] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20240502 08:04:53.705658 25636 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 0.
I20240502 08:04:53.707449 25525 leader_election.cc:302] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:53.708241 25686 raft_consensus.cc:2781] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.708582 25686 raft_consensus.cc:482] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.708930 25686 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.707895 25679 raft_consensus.cc:348] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.714120 25679 raft_consensus.cc:374] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.714458 25679 raft_consensus.cc:729] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.715046 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:53.715620 25679 consensus_queue.cc:260] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
W20240502 08:04:53.716928 25527 leader_election.cc:341] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.718428 25686 raft_consensus.cc:504] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.720052 25681 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.720381 25686 leader_election.cc:288] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:53.721346 25681 ts_tablet_manager.cc:1367] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.020s
I20240502 08:04:53.721346 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:53.722122 25636 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.724400 25681 raft_consensus.cc:348] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.725139 25681 raft_consensus.cc:374] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.725417 25681 raft_consensus.cc:729] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.726193 25681 consensus_queue.cc:260] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.727887 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:53.728395 25681 ts_tablet_manager.cc:1398] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:04:53.729331 25681 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
W20240502 08:04:53.731339 25527 leader_election.cc:341] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.733039 25679 ts_tablet_manager.cc:1398] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20240502 08:04:53.735177 25681 tablet_bootstrap.cc:654] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.737766 25636 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 1.
I20240502 08:04:53.742700 25525 leader_election.cc:302] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:53.738118 25679 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.748679 25686 raft_consensus.cc:2781] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.749485 25686 raft_consensus.cc:686] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:53.750705 25686 consensus_queue.cc:237] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.751983 25679 tablet_bootstrap.cc:654] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.760272 25681 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.760696 25681 ts_tablet_manager.cc:1367] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.016s
I20240502 08:04:53.762728 25681 raft_consensus.cc:348] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.763455 25681 raft_consensus.cc:374] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.763757 25681 raft_consensus.cc:729] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.764505 25681 consensus_queue.cc:260] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.766873 25681 ts_tablet_manager.cc:1398] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 08:04:53.765911 25382 catalog_manager.cc:5484] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:53.767825 25681 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:53.773860 25687 raft_consensus.cc:482] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.774391 25687 raft_consensus.cc:504] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.776242 25681 tablet_bootstrap.cc:654] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.782763 25679 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:53.783236 25679 ts_tablet_manager.cc:1367] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.000s
I20240502 08:04:53.776981 25687 leader_election.cc:288] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:53.791255 25680 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:53.791734 25680 ts_tablet_manager.cc:1367] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.004s
I20240502 08:04:53.785845 25679 raft_consensus.cc:348] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:53.793499 25679 raft_consensus.cc:374] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.793820 25679 raft_consensus.cc:729] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.794553 25679 consensus_queue.cc:260] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:53.799783 25680 raft_consensus.cc:348] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.800729 25680 raft_consensus.cc:374] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.801101 25680 raft_consensus.cc:729] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.801879 25680 consensus_queue.cc:260] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.804062 25679 ts_tablet_manager.cc:1398] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240502 08:04:53.806727 25680 ts_tablet_manager.cc:1398] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20240502 08:04:53.807916 25680 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:53.812168 25679 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.815039 25680 tablet_bootstrap.cc:654] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.818678 25679 tablet_bootstrap.cc:654] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.820344 25685 raft_consensus.cc:482] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.821010 25685 raft_consensus.cc:504] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.827620 25680 tablet_bootstrap.cc:492] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:53.828110 25680 ts_tablet_manager.cc:1367] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20240502 08:04:53.828729 25679 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:53.829279 25679 ts_tablet_manager.cc:1367] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.002s
I20240502 08:04:53.828235 25685 leader_election.cc:288] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:53.832152 25679 raft_consensus.cc:348] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.833037 25679 raft_consensus.cc:374] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.833323 25679 raft_consensus.cc:729] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.831504 25680 raft_consensus.cc:348] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.834753 25680 raft_consensus.cc:374] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.840111 25687 raft_consensus.cc:482] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.841830 25687 raft_consensus.cc:504] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.843760 25687 leader_election.cc:288] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:53.834671 25679 consensus_queue.cc:260] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.840242 25680 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.849589 25680 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.852589 25680 ts_tablet_manager.cc:1398] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20240502 08:04:53.853776 25680 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:53.855224 25681 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.863575 25681 ts_tablet_manager.cc:1367] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.096s user 0.013s sys 0.003s
I20240502 08:04:53.848703 25679 ts_tablet_manager.cc:1398] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.019s user 0.004s sys 0.004s
I20240502 08:04:53.867769 25681 raft_consensus.cc:348] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.868973 25681 raft_consensus.cc:374] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.870007 25681 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.869567 25679 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:53.870702 25681 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.873036 25685 raft_consensus.cc:482] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.873488 25685 raft_consensus.cc:504] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.875506 25685 leader_election.cc:288] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:53.882855 25681 ts_tablet_manager.cc:1398] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20240502 08:04:53.892729 25681 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:53.892369 25679 tablet_bootstrap.cc:654] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.901654 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:53.901571 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:53.906847 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:53.907460 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:53.908502 25486 raft_consensus.cc:2454] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:53.907701 25487 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:53.909660 25681 tablet_bootstrap.cc:654] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.910434 25601 leader_election.cc:302] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:53.913925 25601 leader_election.cc:302] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:53.914907 25686 raft_consensus.cc:482] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.914829 25687 raft_consensus.cc:2781] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.919265 25687 raft_consensus.cc:482] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.918740 25686 raft_consensus.cc:504] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.919710 25687 raft_consensus.cc:3037] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.924948 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:53.926656 25561 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:53.926752 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:53.927006 25686 leader_election.cc:288] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
W20240502 08:04:53.919649 25450 leader_election.cc:341] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.930197 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:53.933287 25686 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.934724 25487 raft_consensus.cc:2454] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 0.
I20240502 08:04:53.925380 25680 tablet_bootstrap.cc:654] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
W20240502 08:04:53.929543 25599 leader_election.cc:341] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.934684 25686 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.935091 25685 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:04:53.935313 25450 leader_election.cc:341] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.936355 25527 leader_election.cc:302] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:04:53.936511 25685 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:53.938436 25688 raft_consensus.cc:2781] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.938951 25688 raft_consensus.cc:482] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.939152 25686 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:53.942736 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:53.940872 25688 raft_consensus.cc:3037] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.939886 25685 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:53.945242 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:53.946302 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
W20240502 08:04:53.948088 25527 leader_election.cc:341] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.952100 25681 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:53.952610 25681 ts_tablet_manager.cc:1367] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.013s
I20240502 08:04:53.952206 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:53.953578 25635 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 0.
W20240502 08:04:53.947657 25450 leader_election.cc:341] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.957003 25687 raft_consensus.cc:504] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.958266 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:53.978652 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:53.973873 25679 tablet_bootstrap.cc:492] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
W20240502 08:04:53.973106 25525 leader_election.cc:341] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.980803 25636 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 0.
I20240502 08:04:53.981063 25681 raft_consensus.cc:348] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.973758 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:53.981915 25680 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:53.981882 25681 raft_consensus.cc:374] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.981560 25525 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:53.982281 25715 raft_consensus.cc:2781] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.983534 25680 ts_tablet_manager.cc:1367] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.130s user 0.018s sys 0.004s
I20240502 08:04:53.982904 25679 ts_tablet_manager.cc:1367] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.114s user 0.020s sys 0.008s
I20240502 08:04:53.983424 25450 leader_election.cc:302] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:53.986449 25686 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.986829 25686 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.984098 25687 leader_election.cc:288] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:53.983966 25681 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.987722 25686 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.987406 25715 raft_consensus.cc:482] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.990247 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:53.990343 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:53.988057 25685 raft_consensus.cc:2781] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.995543 25487 raft_consensus.cc:3037] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.995545 25685 raft_consensus.cc:482] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.995337 25679 raft_consensus.cc:348] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.998870 25680 raft_consensus.cc:348] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.989149 25635 raft_consensus.cc:2379] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e14b3f66048a41d5bfe92033b1925327 in current term 1: Already voted for candidate f03e6bea2e5c4e91a4779abc8a57d01b in this term.
W20240502 08:04:53.989084 25450 leader_election.cc:341] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:04:54.000855 25599 leader_election.cc:341] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.001312 25450 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327; no voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.000144 25685 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.000463 25680 raft_consensus.cc:374] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.000404 25679 raft_consensus.cc:374] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.999075 25688 raft_consensus.cc:504] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:53.999789 25715 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.001459 25681 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.003463 25680 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.006690 25680 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.003242 25679 raft_consensus.cc:729] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.009325 25679 consensus_queue.cc:260] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.014647 25686 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.018321 25685 raft_consensus.cc:504] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.023187 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.025178 25486 raft_consensus.cc:3037] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.027985 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.030503 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
W20240502 08:04:54.031024 25527 leader_election.cc:341] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.029062 25688 leader_election.cc:288] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.031880 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:54.036990 25685 leader_election.cc:288] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.037655 25685 raft_consensus.cc:2726] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.039171 25679 ts_tablet_manager.cc:1398] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.053s user 0.006s sys 0.003s
W20240502 08:04:54.039083 25450 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:34147, user_credentials={real_user=slave}} blocked reactor thread for 50362us
I20240502 08:04:54.040135 25679 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:54.041581 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:54.041826 25450 leader_election.cc:302] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327; no voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.042244 25634 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.043679 25681 ts_tablet_manager.cc:1398] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.090s user 0.011s sys 0.000s
I20240502 08:04:54.045099 25681 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:54.049324 25679 tablet_bootstrap.cc:654] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.051626 25634 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 1.
I20240502 08:04:54.052978 25450 leader_election.cc:302] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.029067 25686 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.053961 25635 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:04:54.031908 25525 leader_election.cc:341] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.057121 25681 tablet_bootstrap.cc:654] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.058799 25720 raft_consensus.cc:2781] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.059525 25720 raft_consensus.cc:686] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:54.060679 25720 consensus_queue.cc:237] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.067490 25487 raft_consensus.cc:2454] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.071058 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:54.072496 25635 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 1.
W20240502 08:04:54.072610 25450 leader_election.cc:341] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.074021 25525 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.070744 25685 raft_consensus.cc:2726] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.075138 25686 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.075346 25601 leader_election.cc:302] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.075631 25686 raft_consensus.cc:686] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:54.077065 25687 raft_consensus.cc:2781] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.077440 25680 ts_tablet_manager.cc:1398] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.092s user 0.008s sys 0.011s
I20240502 08:04:54.077714 25686 consensus_queue.cc:237] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.079811 25680 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:54.081036 25486 raft_consensus.cc:2454] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 1.
I20240502 08:04:54.082784 25527 leader_election.cc:302] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters: f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.087143 25715 raft_consensus.cc:504] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.094516 25679 tablet_bootstrap.cc:492] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:54.095235 25679 ts_tablet_manager.cc:1367] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.015s
I20240502 08:04:54.092886 25687 raft_consensus.cc:686] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:54.106801 25687 consensus_queue.cc:237] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.109951 25719 raft_consensus.cc:2781] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.111073 25719 raft_consensus.cc:686] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:54.112257 25679 raft_consensus.cc:348] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.113257 25679 raft_consensus.cc:374] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.114296 25679 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.114373 25715 leader_election.cc:288] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:54.115141 25680 tablet_bootstrap.cc:654] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.115260 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.116036 25486 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.116217 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:54.116993 25562 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.112825 25719 consensus_queue.cc:237] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.118642 25679 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.122740 25681 tablet_bootstrap.cc:492] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:54.123210 25681 ts_tablet_manager.cc:1367] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.078s user 0.019s sys 0.009s
I20240502 08:04:54.126394 25679 ts_tablet_manager.cc:1398] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.031s user 0.012s sys 0.001s
I20240502 08:04:54.127360 25679 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:54.128906 25486 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.130565 25562 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.139945 25599 leader_election.cc:302] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.142112 25715 raft_consensus.cc:2781] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.142611 25715 raft_consensus.cc:686] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:54.143448 25715 consensus_queue.cc:237] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.152961 25381 catalog_manager.cc:5484] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 0 to 1, leader changed from <none> to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 1 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.154829 25681 raft_consensus.cc:348] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.161223 25681 raft_consensus.cc:374] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.161557 25681 raft_consensus.cc:729] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.162290 25681 consensus_queue.cc:260] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.164849 25681 ts_tablet_manager.cc:1398] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20240502 08:04:54.158870 25679 tablet_bootstrap.cc:654] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.158629 25384 catalog_manager.cc:5484] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 0 to 1, leader changed from <none> to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 1 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:54.171311 25680 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:54.171883 25680 ts_tablet_manager.cc:1367] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.024s
I20240502 08:04:54.180832 25681 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:54.187229 25681 tablet_bootstrap.cc:654] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.184551 25680 raft_consensus.cc:348] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.189735 25680 raft_consensus.cc:374] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.190176 25680 raft_consensus.cc:729] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.191031 25680 consensus_queue.cc:260] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.194828 25680 ts_tablet_manager.cc:1398] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20240502 08:04:54.195781 25680 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:54.196885 25715 raft_consensus.cc:482] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.188737 25383 catalog_manager.cc:5484] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.197713 25715 raft_consensus.cc:504] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.199527 25383 catalog_manager.cc:5484] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.202369 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.203024 25715 leader_election.cc:288] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:54.203511 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010bb756381048b98401bce75b15f03b" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:54.204488 25486 raft_consensus.cc:2379] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f03e6bea2e5c4e91a4779abc8a57d01b in current term 1: Already voted for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in this term.
I20240502 08:04:54.205574 25681 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:54.206130 25681 ts_tablet_manager.cc:1367] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.004s
I20240502 08:04:54.206840 25679 tablet_bootstrap.cc:492] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:54.206686 25601 leader_election.cc:302] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03e6bea2e5c4e91a4779abc8a57d01b; no voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.207394 25679 ts_tablet_manager.cc:1367] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.006s
I20240502 08:04:54.207851 25715 raft_consensus.cc:3037] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.209409 25687 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.211220 25687 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.212937 25680 tablet_bootstrap.cc:654] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.213069 25687 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:54.214172 25679 raft_consensus.cc:348] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.214332 25681 raft_consensus.cc:348] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.214857 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:54.215078 25681 raft_consensus.cc:374] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.215330 25679 raft_consensus.cc:374] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.215478 25681 raft_consensus.cc:729] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.216022 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.217207 25681 consensus_queue.cc:260] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.216832 25679 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.215538 25562 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:54.219386 25679 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.221932 25599 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.223102 25687 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:54.223057 25486 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:54.223623 25687 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:54.224023 25687 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.234002 25679 ts_tablet_manager.cc:1398] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20240502 08:04:54.234021 25681 ts_tablet_manager.cc:1398] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.028s user 0.002s sys 0.004s
I20240502 08:04:54.235257 25679 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: Bootstrap starting.
I20240502 08:04:54.235801 25681 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: Bootstrap starting.
I20240502 08:04:54.237460 25486 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.238651 25687 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.240705 25687 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:54.240949 25679 tablet_bootstrap.cc:654] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.241482 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.242156 25487 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.243340 25680 tablet_bootstrap.cc:492] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:54.243800 25680 ts_tablet_manager.cc:1367] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.008s
I20240502 08:04:54.247246 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:54.247159 25680 raft_consensus.cc:348] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.247848 25562 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.248070 25680 raft_consensus.cc:374] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.248422 25680 raft_consensus.cc:729] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.249236 25680 consensus_queue.cc:260] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.250279 25687 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.250746 25687 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.252631 25687 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785)
I20240502 08:04:54.253427 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
W20240502 08:04:54.254810 25599 leader_election.cc:341] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.255004 25680 ts_tablet_manager.cc:1398] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20240502 08:04:54.255973 25680 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
I20240502 08:04:54.256564 25679 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: No bootstrap required, opened a new log
I20240502 08:04:54.256477 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.257061 25679 ts_tablet_manager.cc:1367] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20240502 08:04:54.257215 25485 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 0.
I20240502 08:04:54.258507 25601 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.259352 25687 raft_consensus.cc:2781] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:54.259697 25687 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:54.260038 25687 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.260054 25679 raft_consensus.cc:348] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.260910 25679 raft_consensus.cc:374] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.261191 25679 raft_consensus.cc:729] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.230036 25715 raft_consensus.cc:2726] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.261826 25679 consensus_queue.cc:260] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.264084 25679 ts_tablet_manager.cc:1398] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:04:54.265084 25562 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.264103 25487 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.266383 25599 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.269541 25680 tablet_bootstrap.cc:654] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.270211 25715 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.266233 25687 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.270735 25715 raft_consensus.cc:686] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:54.271549 25715 consensus_queue.cc:237] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.272202 25687 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785)
I20240502 08:04:54.275650 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.276301 25487 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.283078 25487 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 1.
I20240502 08:04:54.278380 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
W20240502 08:04:54.284693 25599 leader_election.cc:341] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:54.285670 25384 catalog_manager.cc:5484] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 0 to 1, leader changed from <none> to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 1 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:54.286466 25601 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters: c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.287554 25687 raft_consensus.cc:2781] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.288031 25687 raft_consensus.cc:686] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:54.288828 25720 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.289474 25719 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:54.289558 25720 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.291553 25720 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.288833 25687 consensus_queue.cc:237] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.293478 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:54.296418 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.297212 25635 raft_consensus.cc:2379] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e14b3f66048a41d5bfe92033b1925327 in current term 1: Already voted for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in this term.
I20240502 08:04:54.298835 25450 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327; no voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.299576 25720 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.307804 25720 raft_consensus.cc:2726] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.310647 25685 raft_consensus.cc:482] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.314217 25685 raft_consensus.cc:504] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.317394 25685 leader_election.cc:288] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.312371 25680 tablet_bootstrap.cc:492] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:54.318353 25680 ts_tablet_manager.cc:1367] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.063s user 0.017s sys 0.003s
I20240502 08:04:54.318470 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.312506 25719 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 08:04:54.321496 25680 raft_consensus.cc:348] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.322275 25680 raft_consensus.cc:374] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.322546 25680 raft_consensus.cc:729] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.322508 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:54.323279 25680 consensus_queue.cc:260] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.324208 25450 leader_election.cc:302] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327; no voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.324965 25685 raft_consensus.cc:2726] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:04:54.325249 25680 ts_tablet_manager.cc:1398] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:04:54.337791 25381 catalog_manager.cc:5484] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 0 to 1, leader changed from <none> to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 1 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:54.340466 25381 catalog_manager.cc:5484] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 0 to 1, leader changed from <none> to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 1 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.310002 25681 tablet_bootstrap.cc:654] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.344369 25680 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: Bootstrap starting.
W20240502 08:04:54.350342 25432 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:04:54.368494 25680 tablet_bootstrap.cc:654] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:54.374594 25681 tablet_bootstrap.cc:492] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: No bootstrap required, opened a new log
I20240502 08:04:54.375170 25681 ts_tablet_manager.cc:1367] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent bootstrapping tablet: real 0.140s user 0.007s sys 0.013s
I20240502 08:04:54.377548 25681 raft_consensus.cc:348] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.380302 25681 raft_consensus.cc:374] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.380590 25681 raft_consensus.cc:729] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.381199 25681 consensus_queue.cc:260] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.383154 25681 ts_tablet_manager.cc:1398] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240502 08:04:54.385190 25680 tablet_bootstrap.cc:492] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: No bootstrap required, opened a new log
I20240502 08:04:54.385545 25680 ts_tablet_manager.cc:1367] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.001s
I20240502 08:04:54.387239 25680 raft_consensus.cc:348] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.387833 25680 raft_consensus.cc:374] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:54.388124 25680 raft_consensus.cc:729] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Initialized, Role: FOLLOWER
I20240502 08:04:54.388707 25680 consensus_queue.cc:260] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.390725 25680 ts_tablet_manager.cc:1398] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240502 08:04:54.400151 25686 raft_consensus.cc:482] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.400651 25686 raft_consensus.cc:504] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.402943 25686 leader_election.cc:288] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.404266 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.404870 25634 raft_consensus.cc:2379] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9a7b014b4a643818eb8f3cbf44b69e8 in current term 1: Already voted for candidate e14b3f66048a41d5bfe92033b1925327 in this term.
I20240502 08:04:54.405615 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.406769 25527 leader_election.cc:302] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8; no voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.407449 25686 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.411587 25686 raft_consensus.cc:2726] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.484990 25685 raft_consensus.cc:482] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.485574 25685 raft_consensus.cc:504] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.487648 25685 leader_election.cc:288] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.488451 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" is_pre_election: true
I20240502 08:04:54.488631 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.489187 25634 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 0.
I20240502 08:04:54.489123 25562 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 0.
I20240502 08:04:54.490193 25450 leader_election.cc:302] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.490926 25685 raft_consensus.cc:2781] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:54.491226 25685 raft_consensus.cc:482] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:54.491504 25685 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.495522 25685 raft_consensus.cc:504] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.497133 25685 leader_election.cc:288] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.498276 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:54.498364 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:54.498934 25562 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.499362 25634 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.503484 25562 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 1.
I20240502 08:04:54.504374 25450 leader_election.cc:302] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:04:54.505208 25685 raft_consensus.cc:2781] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.505534 25634 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 1.
I20240502 08:04:54.506435 25685 raft_consensus.cc:686] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:54.507072 25685 consensus_queue.cc:237] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.514333 25383 catalog_manager.cc:5484] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 0 to 1, leader changed from <none> to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 1 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.521350 25686 consensus_queue.cc:1035] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.532738 25715 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.534361 25688 consensus_queue.cc:1035] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.555341 25722 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:54.580308 25688 raft_consensus.cc:482] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.580725 25688 raft_consensus.cc:504] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.582269 25688 leader_election.cc:288] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.583127 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.583309 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd0117fac064eaea657db7a0263b0a2" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.583743 25486 raft_consensus.cc:2379] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9a7b014b4a643818eb8f3cbf44b69e8 in current term 1: Already voted for candidate f03e6bea2e5c4e91a4779abc8a57d01b in this term.
I20240502 08:04:54.584805 25527 leader_election.cc:302] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8; no voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.585453 25688 raft_consensus.cc:3037] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.590189 25722 consensus_queue.cc:1035] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.591951 25688 raft_consensus.cc:2726] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:54.603719 25715 consensus_queue.cc:1035] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.607770 25688 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.618808 25688 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.687973 25685 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.689458 25722 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.707551 25687 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.711431 25715 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.711652 25720 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.737375 25562 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.749655 25688 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:54.750198 25688 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:54.750566 25687 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.752849 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" is_pre_election: true
I20240502 08:04:54.753552 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14b3f66048a41d5bfe92033b1925327" is_pre_election: true
I20240502 08:04:54.755844 25527 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8; no voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:54.754401 25688 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147), e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785)
I20240502 08:04:54.756587 25686 raft_consensus.cc:2726] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:04:54.781443 25634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:60034
I20240502 08:04:54.782250 25634 raft_consensus.cc:595] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.796556 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:46056
I20240502 08:04:54.797076 25562 raft_consensus.cc:595] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.802228 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:46056
I20240502 08:04:54.802839 25562 raft_consensus.cc:595] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.807588 25634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:60034
I20240502 08:04:54.808187 25634 raft_consensus.cc:595] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:04:54.813002 25634 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:60034
I20240502 08:04:54.813566 25634 raft_consensus.cc:595] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.828716 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:43820
I20240502 08:04:54.829219 25485 raft_consensus.cc:595] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.834211 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:43820
I20240502 08:04:54.834751 25485 raft_consensus.cc:595] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:04:54.883551 25688 raft_consensus.cc:979] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:54.884027 25686 raft_consensus.cc:1067] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:54.885676 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43804
I20240502 08:04:54.886214 25485 raft_consensus.cc:482] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:54.886492 25485 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.890868 25485 raft_consensus.cc:504] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.892395 25485 leader_election.cc:288] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.893261 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:54.893591 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:54.893877 25562 raft_consensus.cc:3032] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:54.894176 25634 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.894268 25562 raft_consensus.cc:729] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:54.895117 25562 consensus_queue.cc:260] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.896493 25562 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.900635 25634 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:54.901748 25450 leader_election.cc:302] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:54.902640 25685 raft_consensus.cc:2781] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:54.902751 25562 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:54.903244 25685 raft_consensus.cc:686] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:54.904096 25685 consensus_queue.cc:237] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.911162 25381 catalog_manager.cc:5484] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 1 to 2, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.936414 25685 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.943616 25720 raft_consensus.cc:979] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:54.944262 25685 raft_consensus.cc:1067] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:54.946630 25561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46052
I20240502 08:04:54.947842 25561 raft_consensus.cc:482] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:54.948235 25561 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.951879 25685 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:54.960201 25561 raft_consensus.cc:504] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.964020 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:54.964823 25485 raft_consensus.cc:3032] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:54.965222 25485 raft_consensus.cc:729] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:54.966213 25485 consensus_queue.cc:260] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.967397 25485 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.972506 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:54.973127 25634 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:54.974968 25485 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:54.976097 25527 leader_election.cc:302] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:04:54.977049 25561 leader_election.cc:288] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:54.977392 25686 raft_consensus.cc:2781] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:54.977898 25686 raft_consensus.cc:686] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:54.978736 25686 consensus_queue.cc:237] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:54.979619 25634 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:54.986691 25382 catalog_manager.cc:5484] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 1 to 2, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.094342 25722 raft_consensus.cc:979] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.094767 25687 raft_consensus.cc:1067] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.096103 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43818
I20240502 08:04:55.096683 25485 raft_consensus.cc:482] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:55.097064 25485 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.103623 25485 raft_consensus.cc:504] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.106355 25485 leader_election.cc:288] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:55.107724 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:55.108206 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:55.108410 25561 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.108794 25635 raft_consensus.cc:3032] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:55.109107 25635 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:55.109839 25635 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.111083 25635 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.115679 25561 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.116811 25450 leader_election.cc:302] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:04:55.117622 25764 raft_consensus.cc:2781] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.118084 25764 raft_consensus.cc:686] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:55.118870 25764 consensus_queue.cc:237] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.120397 25635 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.128110 25382 catalog_manager.cc:5484] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 1 to 2, leader changed from f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.227852 25722 raft_consensus.cc:979] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:55.228340 25687 raft_consensus.cc:1067] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:55.229892 25561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46036
I20240502 08:04:55.230504 25561 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:55.230890 25561 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.235853 25722 raft_consensus.cc:979] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.236336 25687 raft_consensus.cc:1067] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.236619 25561 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.238469 25561 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:55.238461 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43818
I20240502 08:04:55.239154 25485 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:55.239540 25485 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.240622 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:55.241191 25486 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.243486 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:55.244100 25635 raft_consensus.cc:3032] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:55.244405 25635 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:55.245548 25686 raft_consensus.cc:979] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.245123 25485 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:55.246117 25686 raft_consensus.cc:1067] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:04:55.245605 25635 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.247782 25635 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.248610 25485 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Requested vote from peers f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:55.248759 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:55.249637 25634 raft_consensus.cc:3032] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:55.250123 25634 raft_consensus.cc:729] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:55.251188 25634 consensus_queue.cc:260] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:55.252734 25634 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.254510 25635 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:55.255115 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:55.256003 25561 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.256076 25525 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:55.257378 25688 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.257992 25688 raft_consensus.cc:686] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:55.258893 25688 consensus_queue.cc:237] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.260490 25486 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:55.264344 25561 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.265934 25450 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:04:55.266916 25764 raft_consensus.cc:2781] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.267381 25764 raft_consensus.cc:686] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:55.267730 25634 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.268193 25764 consensus_queue.cc:237] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:04:55.269667 25382 catalog_manager.cc:5484] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 1 to 2, leader changed from f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.270931 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43804
I20240502 08:04:55.271548 25485 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:55.273895 25485 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.277220 25381 catalog_manager.cc:5484] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 1 to 2, leader changed from f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:55.280150 25485 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.283447 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:55.284024 25561 raft_consensus.cc:3032] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:55.284091 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:55.284415 25561 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:55.284687 25634 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.285177 25561 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.286301 25561 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.287274 25485 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:55.290382 25634 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.291285 25450 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:55.291908 25764 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.292285 25764 raft_consensus.cc:686] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:55.293375 25561 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 2.
I20240502 08:04:55.293529 25764 consensus_queue.cc:237] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.301000 25382 catalog_manager.cc:5484] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 1 to 2, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.321321 25720 raft_consensus.cc:979] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:55.321739 25764 raft_consensus.cc:1067] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:04:55.323060 25561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46052
I20240502 08:04:55.323518 25561 raft_consensus.cc:482] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:55.323801 25561 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.328015 25561 raft_consensus.cc:504] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.329476 25561 leader_election.cc:288] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:04:55.330233 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:55.330432 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:04:55.330752 25485 raft_consensus.cc:3032] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:55.330907 25634 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.331053 25485 raft_consensus.cc:729] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:55.331714 25485 consensus_queue.cc:260] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.332556 25485 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.335340 25634 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:55.336266 25525 leader_election.cc:302] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:55.337039 25688 raft_consensus.cc:2781] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.337393 25688 raft_consensus.cc:686] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:55.337455 25485 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 2.
I20240502 08:04:55.338177 25688 consensus_queue.cc:237] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:55.344390 25382 catalog_manager.cc:5484] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 1 to 2, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.442828 25634 raft_consensus.cc:1261] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.444053 25688 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.452337 25485 raft_consensus.cc:1261] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.453682 25688 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.457190 25634 raft_consensus.cc:1261] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.458691 25764 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.470327 25561 raft_consensus.cc:1261] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.472083 25685 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.566326 25561 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.567612 25685 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.575371 25634 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.576912 25720 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.723788 25485 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.724989 25688 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.733219 25634 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.734810 25686 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.763731 25485 raft_consensus.cc:1261] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.764819 25686 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.772531 25634 raft_consensus.cc:1261] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.774020 25719 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.809419 25561 raft_consensus.cc:1261] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.810590 25720 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.817921 25634 raft_consensus.cc:1261] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.819307 25720 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:04:55.834903 25561 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.836042 25720 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.844105 25634 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.845443 25685 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:57.863616 25352 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240502 08:04:57.864804 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:57.890365 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:44362
I20240502 08:04:57.890832 25485 raft_consensus.cc:595] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Received request to transfer leadership to TS f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:57.900542 25561 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:04:57.901049 25561 raft_consensus.cc:595] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Received request to transfer leadership to TS f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:04:57.902114 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 08:04:57.902547 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:57.979573 25685 raft_consensus.cc:979] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: : Instructing follower f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:04:57.979974 25685 raft_consensus.cc:1067] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Signalling peer f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:04:57.981194 25634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:60032
I20240502 08:04:57.981643 25634 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:57.981938 25634 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:57.986344 25634 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:57.987742 25634 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:57.988633 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:57.988696 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:57.989167 25485 raft_consensus.cc:3032] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:04:57.989261 25561 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:57.989442 25485 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:04:57.990175 25485 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:57.991022 25485 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:57.993564 25561 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:04:57.994542 25599 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:57.995152 25781 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:04:57.995251 25485 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:04:57.996412 25781 raft_consensus.cc:686] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:57.997143 25781 consensus_queue.cc:237] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:58.005816 25381 catalog_manager.cc:5484] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 2 to 3, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 3 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:58.284961 25686 raft_consensus.cc:979] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:04:58.285365 25792 raft_consensus.cc:1067] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Signalling peer f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:04:58.286690 25634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:60028
I20240502 08:04:58.287124 25634 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:58.287371 25634 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:58.290992 25634 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:58.292261 25634 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:04:58.293123 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:04:58.293330 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:04:58.293733 25485 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:58.293840 25561 raft_consensus.cc:3032] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:04:58.294195 25561 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:04:58.294909 25561 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:58.296094 25561 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:58.299973 25485 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:04:58.300937 25601 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:04:58.301571 25781 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:04:58.302105 25781 raft_consensus.cc:686] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:04:58.302181 25561 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:04:58.303005 25781 consensus_queue.cc:237] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:04:58.309172 25381 catalog_manager.cc:5484] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 2 to 3, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 3 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:58.471937 25485 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:58.473237 25781 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:58.481840 25561 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:58.483189 25796 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:58.733604 25485 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:58.734989 25781 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:58.736304 25561 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:58.737591 25781 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:58.903424 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:58.909575 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:58.909970 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:59.910764 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:59.917359 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:59.917755 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:00.918641 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:00.924279 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:00.924791 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:01.925896 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:01.930897 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:01.931277 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:02.931988 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:02.936802 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:02.937150 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:03.937814 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:03.942549 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:03.942902 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:04.943629 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:04.948837 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:04.949183 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:05.949896 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:05.955267 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:05.955807 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:06.956558 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:06.962917 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:06.963366 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:07.986234 25352 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240502 08:05:07.990260 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:43820
I20240502 08:05:07.990921 25485 raft_consensus.cc:595] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:05:07.995707 25635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:60034
I20240502 08:05:07.996215 25635 raft_consensus.cc:595] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:05:08.000341 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:43820
I20240502 08:05:08.000779 25485 raft_consensus.cc:595] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:05:08.004979 25485 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:43820
I20240502 08:05:08.005414 25485 raft_consensus.cc:595] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:05:08.009438 25635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
mode: GRACEFUL
new_leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:60034
I20240502 08:05:08.009956 25635 raft_consensus.cc:595] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Received request to transfer leadership to TS c9a7b014b4a643818eb8f3cbf44b69e8
I20240502 08:05:08.088851 25810 raft_consensus.cc:979] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.089380 25804 raft_consensus.cc:1067] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.090754 25562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46036
I20240502 08:05:08.091199 25562 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:08.091514 25562 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.096141 25562 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.097663 25562 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 4 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:08.098510 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:08.098584 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:08.099264 25485 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.099357 25635 raft_consensus.cc:3032] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:08.099696 25635 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:05:08.100346 25635 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.101310 25635 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.103789 25485 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 4.
I20240502 08:05:08.104889 25527 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:05:08.105814 25809 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:08.106398 25635 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 4.
I20240502 08:05:08.106726 25809 raft_consensus.cc:686] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:08.107560 25809 consensus_queue.cc:237] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.115473 25382 catalog_manager.cc:5484] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 3 to 4, leader changed from f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 4 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:08.155902 25806 raft_consensus.cc:979] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.156422 25813 raft_consensus.cc:1067] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.157804 25562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46052
I20240502 08:05:08.158288 25562 raft_consensus.cc:482] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:08.158572 25562 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.162619 25562 raft_consensus.cc:504] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.163913 25562 leader_election.cc:288] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:08.164798 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:08.164906 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:08.165262 25485 raft_consensus.cc:3032] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:08.165395 25635 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.165522 25485 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:08.166199 25485 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.167079 25485 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.170522 25635 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.171454 25525 leader_election.cc:302] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:08.171911 25485 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.172044 25809 raft_consensus.cc:2781] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:08.172508 25809 raft_consensus.cc:686] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:08.173331 25809 consensus_queue.cc:237] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.180176 25382 catalog_manager.cc:5484] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 2 to 3, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 3 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:08.192386 25806 raft_consensus.cc:979] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.192757 25813 raft_consensus.cc:1067] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.193876 25562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46052
I20240502 08:05:08.194308 25562 raft_consensus.cc:482] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:08.194521 25562 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.198128 25562 raft_consensus.cc:504] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.199419 25562 leader_election.cc:288] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:08.200042 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:08.200230 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478eeaef42cf494eb349a9ca2a4d7b17" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:08.200592 25485 raft_consensus.cc:3032] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:08.200752 25635 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.200925 25485 raft_consensus.cc:729] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:08.201664 25485 consensus_queue.cc:260] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.202845 25485 raft_consensus.cc:3037] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.206861 25635 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.207801 25525 leader_election.cc:302] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:08.208480 25809 raft_consensus.cc:2781] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:08.208647 25485 raft_consensus.cc:2454] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.208855 25809 raft_consensus.cc:686] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:08.209712 25809 consensus_queue.cc:237] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.216295 25382 catalog_manager.cc:5484] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 2 to 3, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 3 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:08.315764 25810 raft_consensus.cc:979] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.316169 25804 raft_consensus.cc:1067] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.317399 25562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46036
I20240502 08:05:08.317816 25562 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:08.318075 25562 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.321863 25562 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.323391 25562 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 4 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:08.324060 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:08.324322 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:08.324509 25485 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.324827 25635 raft_consensus.cc:3032] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:08.325083 25635 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:05:08.325713 25635 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.326674 25635 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:08.328377 25485 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 4.
I20240502 08:05:08.329141 25527 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:05:08.329735 25809 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:08.330130 25809 raft_consensus.cc:686] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:08.331104 25635 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 4.
I20240502 08:05:08.330889 25809 consensus_queue.cc:237] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:08.337404 25382 catalog_manager.cc:5484] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 3 to 4, leader changed from f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 4 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:08.347079 25806 raft_consensus.cc:979] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: : Instructing follower c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.347482 25813 raft_consensus.cc:1067] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Signalling peer c9a7b014b4a643818eb8f3cbf44b69e8 to start an election
I20240502 08:05:08.348731 25562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
from {username='slave'} at 127.0.0.1:46052
I20240502 08:05:08.349123 25562 raft_consensus.cc:482] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:08.349347 25562 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.353403 25562 raft_consensus.cc:504] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:05:08.354789 25562 leader_election.cc:288] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Requested vote from peers f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147), e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785)
I20240502 08:05:08.355655 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:08.355620 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a7530ae84444f28528e2ecc8d1ed96" candidate_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:08.356305 25485 raft_consensus.cc:3032] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:08.356362 25635 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.356582 25485 raft_consensus.cc:729] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:08.357199 25485 consensus_queue.cc:260] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:05:08.358100 25485 raft_consensus.cc:3037] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:08.360792 25635 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.361634 25525 leader_election.cc:302] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:08.361941 25485 raft_consensus.cc:2454] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7b014b4a643818eb8f3cbf44b69e8 in term 3.
I20240502 08:05:08.362311 25809 raft_consensus.cc:2781] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:08.362752 25809 raft_consensus.cc:686] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Becoming Leader. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:08.363449 25809 consensus_queue.cc:237] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } }
I20240502 08:05:08.369846 25382 catalog_manager.cc:5484] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 reported cstate change: term changed from 2 to 3, leader changed from e14b3f66048a41d5bfe92033b1925327 (127.24.194.1) to c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2). New cstate: current_term: 3 leader_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:08.525229 25635 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:08.526393 25809 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:08.536227 25485 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:08.538357 25811 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:08.595487 25635 raft_consensus.cc:1261] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.596611 25811 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:08.605897 25485 raft_consensus.cc:1261] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.607296 25811 consensus_queue.cc:1035] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:08.686323 25635 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.687434 25811 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:08.695299 25485 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.696743 25828 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:05:08.754474 25635 raft_consensus.cc:1261] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.755600 25809 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:08.765033 25485 raft_consensus.cc:1261] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:08.766266 25809 consensus_queue.cc:1035] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:08.921162 25635 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:08.922235 25809 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:08.929913 25485 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Refusing update from remote peer c9a7b014b4a643818eb8f3cbf44b69e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:08.931936 25811 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:05:10.765409 25508 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec060 after lost signal to thread 25355
W20240502 08:05:10.766188 25508 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec740 after lost signal to thread 25418
W20240502 08:05:10.767035 25508 debug-util.cc:398] Leaking SignalData structure 0x7b08001a9c80 after lost signal to thread 25511
W20240502 08:05:10.767750 25508 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6d20 after lost signal to thread 25586
W20240502 08:05:10.768359 25508 debug-util.cc:398] Leaking SignalData structure 0x7b08001b8820 after lost signal to thread 25660
I20240502 08:05:11.078336 25352 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240502 08:05:11.079170 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:11.086015 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:05:11.086897 25562 raft_consensus.cc:595] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:05:11.089493 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:05:11.089972 25562 raft_consensus.cc:595] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:05:11.092054 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:05:11.092497 25562 raft_consensus.cc:595] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Received request to transfer leadership to TS e14b3f66048a41d5bfe92033b1925327
I20240502 08:05:11.094513 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:05:11.094944 25562 raft_consensus.cc:595] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Received request to transfer leadership to TS f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:05:11.096840 25562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
mode: GRACEFUL
new_leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:52800
I20240502 08:05:11.097244 25562 raft_consensus.cc:595] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Received request to transfer leadership to TS f03e6bea2e5c4e91a4779abc8a57d01b
I20240502 08:05:11.098138 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 08:05:11.098551 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:11.124253 25828 raft_consensus.cc:979] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:05:11.124830 25838 raft_consensus.cc:1067] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Signalling peer f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:05:11.126689 25635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:60028
I20240502 08:05:11.127336 25635 raft_consensus.cc:482] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:11.127810 25635 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.136854 25635 raft_consensus.cc:504] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.140069 25635 leader_election.cc:288] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 5 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:05:11.141741 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:11.142298 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f3937efc374225b60593697f9d6f79" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:05:11.142601 25485 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.142920 25562 raft_consensus.cc:3032] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:05:11.143256 25562 raft_consensus.cc:729] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:11.144326 25562 consensus_queue.cc:260] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.146358 25562 raft_consensus.cc:3037] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.151695 25485 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 5.
I20240502 08:05:11.152938 25601 leader_election.cc:302] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:11.153818 25837 raft_consensus.cc:2781] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:05:11.153954 25562 raft_consensus.cc:2454] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 5.
I20240502 08:05:11.155193 25837 raft_consensus.cc:686] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:05:11.156309 25837 consensus_queue.cc:237] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.174978 25383 catalog_manager.cc:5484] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 4 to 5, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 5 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:11.181032 25828 raft_consensus.cc:979] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.181594 25838 raft_consensus.cc:1067] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.183488 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43804
I20240502 08:05:11.184142 25485 raft_consensus.cc:482] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:11.184492 25485 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.189769 25485 raft_consensus.cc:504] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.192032 25485 leader_election.cc:288] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 5 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:11.193143 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:11.193850 25635 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.194475 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c877e979fbd43fc947a2cc17c3e17cd" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:05:11.195061 25562 raft_consensus.cc:3032] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:05:11.195369 25562 raft_consensus.cc:729] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:11.196118 25562 consensus_queue.cc:260] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.197480 25562 raft_consensus.cc:3037] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:11.198416 25635 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 5.
I20240502 08:05:11.199456 25450 leader_election.cc:302] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:11.201375 25846 raft_consensus.cc:2781] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:05:11.202693 25846 raft_consensus.cc:686] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 5 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:11.203805 25562 raft_consensus.cc:2454] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 5.
I20240502 08:05:11.203682 25846 consensus_queue.cc:237] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.211143 25383 catalog_manager.cc:5484] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 4 to 5, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 5 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:11.265583 25828 raft_consensus.cc:979] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.266024 25838 raft_consensus.cc:1067] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.267511 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43804
I20240502 08:05:11.268046 25485 raft_consensus.cc:482] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:11.268368 25485 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:11.275061 25485 raft_consensus.cc:504] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.277096 25485 leader_election.cc:288] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 4 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:11.277806 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:05:11.278009 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62009aa71d334e60a3ab5c131817fc2e" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:11.278393 25562 raft_consensus.cc:3032] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:11.278476 25635 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:11.278733 25562 raft_consensus.cc:729] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:11.279512 25562 consensus_queue.cc:260] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.280674 25562 raft_consensus.cc:3037] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:11.284606 25635 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 4.
I20240502 08:05:11.285667 25450 leader_election.cc:302] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e14b3f66048a41d5bfe92033b1925327, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:11.286518 25562 raft_consensus.cc:2454] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 4.
I20240502 08:05:11.286496 25846 raft_consensus.cc:2781] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:11.287220 25846 raft_consensus.cc:686] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 4 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:11.288179 25846 consensus_queue.cc:237] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.295810 25383 catalog_manager.cc:5484] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 3 to 4, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 4 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:11.412905 25828 raft_consensus.cc:979] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:05:11.413344 25838 raft_consensus.cc:1067] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Signalling peer f03e6bea2e5c4e91a4779abc8a57d01b to start an election
I20240502 08:05:11.414683 25635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd"
dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
from {username='slave'} at 127.0.0.1:60028
I20240502 08:05:11.415118 25635 raft_consensus.cc:482] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:11.415361 25635 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.419565 25635 raft_consensus.cc:504] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.420893 25635 leader_election.cc:288] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Requested vote from peers e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785), c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813)
I20240502 08:05:11.421768 25485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
I20240502 08:05:11.421926 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f86f3a4a7944c498b4b6bbbb5d606cd" candidate_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:05:11.422614 25562 raft_consensus.cc:3032] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:11.422593 25485 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.423060 25562 raft_consensus.cc:729] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:11.423745 25562 consensus_queue.cc:260] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.424758 25562 raft_consensus.cc:3037] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.429809 25562 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:05:11.429853 25485 raft_consensus.cc:2454] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e6bea2e5c4e91a4779abc8a57d01b in term 3.
I20240502 08:05:11.430830 25599 leader_election.cc:302] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, f03e6bea2e5c4e91a4779abc8a57d01b; no voters:
I20240502 08:05:11.431597 25837 raft_consensus.cc:2781] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:11.432103 25837 raft_consensus.cc:686] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Becoming Leader. State: Replica: f03e6bea2e5c4e91a4779abc8a57d01b, State: Running, Role: LEADER
I20240502 08:05:11.433033 25837 consensus_queue.cc:237] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.440971 25383 catalog_manager.cc:5484] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b reported cstate change: term changed from 2 to 3, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3). New cstate: current_term: 3 leader_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:11.504559 25828 raft_consensus.cc:979] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: : Instructing follower e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.505076 25828 raft_consensus.cc:1067] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Signalling peer e14b3f66048a41d5bfe92033b1925327 to start an election
I20240502 08:05:11.506567 25485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1"
dest_uuid: "e14b3f66048a41d5bfe92033b1925327"
from {username='slave'} at 127.0.0.1:43804
I20240502 08:05:11.507139 25485 raft_consensus.cc:482] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:11.507477 25485 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.514361 25485 raft_consensus.cc:504] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.516243 25485 leader_election.cc:288] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 3 election: Requested vote from peers c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813), f03e6bea2e5c4e91a4779abc8a57d01b (127.24.194.3:34147)
I20240502 08:05:11.517231 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8"
I20240502 08:05:11.517438 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0933e210c5b442f38d9598dcc8cb87b1" candidate_uuid: "e14b3f66048a41d5bfe92033b1925327" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b"
I20240502 08:05:11.517690 25562 raft_consensus.cc:3032] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:11.518054 25635 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.518260 25562 raft_consensus.cc:729] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9a7b014b4a643818eb8f3cbf44b69e8, State: Running, Role: LEADER
I20240502 08:05:11.518836 25562 consensus_queue.cc:260] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.519717 25562 raft_consensus.cc:3037] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:11.523670 25562 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 3.
I20240502 08:05:11.524544 25450 leader_election.cc:302] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9a7b014b4a643818eb8f3cbf44b69e8, e14b3f66048a41d5bfe92033b1925327; no voters:
I20240502 08:05:11.524546 25635 raft_consensus.cc:2454] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14b3f66048a41d5bfe92033b1925327 in term 3.
I20240502 08:05:11.525403 25846 raft_consensus.cc:2781] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:11.525880 25846 raft_consensus.cc:686] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 3 LEADER]: Becoming Leader. State: Replica: e14b3f66048a41d5bfe92033b1925327, State: Running, Role: LEADER
I20240502 08:05:11.526813 25846 consensus_queue.cc:237] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } }
I20240502 08:05:11.534472 25383 catalog_manager.cc:5484] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 reported cstate change: term changed from 2 to 3, leader changed from c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2) to e14b3f66048a41d5bfe92033b1925327 (127.24.194.1). New cstate: current_term: 3 leader_uuid: "e14b3f66048a41d5bfe92033b1925327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:11.548934 25562 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:11.550218 25837 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:11.562723 25485 raft_consensus.cc:1261] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:11.564558 25840 consensus_queue.cc:1035] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:05:11.671568 25635 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:11.672755 25562 raft_consensus.cc:1261] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:11.672953 25846 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:11.674760 25846 consensus_queue.cc:1035] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:11.767822 25635 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:11.769069 25847 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:11.778316 25561 raft_consensus.cc:1261] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:11.780227 25847 consensus_queue.cc:1035] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:11.971280 25486 raft_consensus.cc:1261] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:11.972373 25837 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14b3f66048a41d5bfe92033b1925327" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:11.982513 25561 raft_consensus.cc:1261] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Refusing update from remote peer f03e6bea2e5c4e91a4779abc8a57d01b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:11.983896 25837 consensus_queue.cc:1035] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:12.010795 25635 raft_consensus.cc:1261] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:12.012017 25846 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03e6bea2e5c4e91a4779abc8a57d01b" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:12.022833 25561 raft_consensus.cc:1261] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Refusing update from remote peer e14b3f66048a41d5bfe92033b1925327: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:12.024024 25846 consensus_queue.cc:1035] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7b014b4a643818eb8f3cbf44b69e8" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:12.099336 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:12.104283 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:12.104665 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:13.105456 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:13.110332 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:13.110703 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:14.111502 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:14.116279 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:14.116637 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:15.117432 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:15.122306 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:15.122721 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:16.123479 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:16.128384 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:16.128829 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:17.129665 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:17.134646 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:17.135051 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:18.135807 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:18.141191 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:18.141577 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:19.142443 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:19.147490 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:19.147886 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:20.148772 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:20.153488 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:20.153925 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:21.177955 25352 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240502 08:05:21.178700 25352 tablet_server.cc:178] TabletServer@127.24.194.1:0 shutting down...
I20240502 08:05:21.205802 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:21.206617 25352 tablet_replica.cc:330] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.207736 25352 raft_consensus.cc:2229] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.208490 25352 raft_consensus.cc:2258] T 9f86f3a4a7944c498b4b6bbbb5d606cd P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.210846 25352 tablet_replica.cc:330] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.211264 25352 raft_consensus.cc:2229] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.211606 25352 raft_consensus.cc:2258] T d8f3937efc374225b60593697f9d6f79 P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.213136 25352 tablet_replica.cc:330] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.213682 25352 raft_consensus.cc:2229] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.214114 25352 raft_consensus.cc:2258] T 478eeaef42cf494eb349a9ca2a4d7b17 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.215654 25352 tablet_replica.cc:330] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.216106 25352 raft_consensus.cc:2229] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.216431 25352 raft_consensus.cc:2258] T 010bb756381048b98401bce75b15f03b P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.217821 25352 tablet_replica.cc:330] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.218351 25352 raft_consensus.cc:2229] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 5 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.219167 25352 raft_consensus.cc:2258] T 1c877e979fbd43fc947a2cc17c3e17cd P e14b3f66048a41d5bfe92033b1925327 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.220650 25352 tablet_replica.cc:330] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.221365 25352 raft_consensus.cc:2229] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.221846 25352 raft_consensus.cc:2258] T f2a7530ae84444f28528e2ecc8d1ed96 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.223867 25352 tablet_replica.cc:330] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.224376 25352 raft_consensus.cc:2229] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.224849 25352 raft_consensus.cc:2258] T 9bd0117fac064eaea657db7a0263b0a2 P e14b3f66048a41d5bfe92033b1925327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.226603 25352 tablet_replica.cc:330] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.227063 25352 raft_consensus.cc:2229] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.227792 25352 raft_consensus.cc:2258] T 0933e210c5b442f38d9598dcc8cb87b1 P e14b3f66048a41d5bfe92033b1925327 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.229346 25352 tablet_replica.cc:330] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327: stopping tablet replica
I20240502 08:05:21.229769 25352 raft_consensus.cc:2229] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 4 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.230489 25352 raft_consensus.cc:2258] T 62009aa71d334e60a3ab5c131817fc2e P e14b3f66048a41d5bfe92033b1925327 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240502 08:05:21.315366 25601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.194.1:43785: connect: Connection refused (error 111)
W20240502 08:05:21.320377 25601 consensus_peers.cc:485] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b -> Peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Couldn't send request to peer e14b3f66048a41d5bfe92033b1925327. Status: Network error: Client connection negotiation failed: client connection to 127.24.194.1:43785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:21.321173 25527 consensus_peers.cc:485] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 -> Peer e14b3f66048a41d5bfe92033b1925327 (127.24.194.1:43785): Couldn't send request to peer e14b3f66048a41d5bfe92033b1925327. Status: Network error: Client connection negotiation failed: client connection to 127.24.194.1:43785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:21.337451 25352 tablet_server.cc:194] TabletServer@127.24.194.1:0 shutdown complete.
I20240502 08:05:21.354387 25352 tablet_server.cc:178] TabletServer@127.24.194.2:0 shutting down...
I20240502 08:05:21.379256 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:21.379860 25352 tablet_replica.cc:330] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.380460 25352 raft_consensus.cc:2229] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.380913 25352 raft_consensus.cc:2258] T 0933e210c5b442f38d9598dcc8cb87b1 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.382714 25352 tablet_replica.cc:330] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.383205 25352 raft_consensus.cc:2229] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.383597 25352 raft_consensus.cc:2258] T 9bd0117fac064eaea657db7a0263b0a2 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.385313 25352 tablet_replica.cc:330] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.385856 25352 raft_consensus.cc:2229] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.386363 25352 raft_consensus.cc:2258] T 9f86f3a4a7944c498b4b6bbbb5d606cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.387863 25352 tablet_replica.cc:330] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.388343 25352 raft_consensus.cc:2229] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.388757 25352 raft_consensus.cc:2258] T d8f3937efc374225b60593697f9d6f79 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.390321 25352 tablet_replica.cc:330] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.390761 25352 raft_consensus.cc:2229] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.391163 25352 raft_consensus.cc:2258] T 62009aa71d334e60a3ab5c131817fc2e P c9a7b014b4a643818eb8f3cbf44b69e8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.392560 25352 tablet_replica.cc:330] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.392985 25352 raft_consensus.cc:2229] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.393715 25352 raft_consensus.cc:2258] T 010bb756381048b98401bce75b15f03b P c9a7b014b4a643818eb8f3cbf44b69e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.395390 25352 tablet_replica.cc:330] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.395830 25352 raft_consensus.cc:2229] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.396982 25352 raft_consensus.cc:2258] T f2a7530ae84444f28528e2ecc8d1ed96 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.398496 25352 tablet_replica.cc:330] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.398891 25352 raft_consensus.cc:2229] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.399242 25352 raft_consensus.cc:2258] T 1c877e979fbd43fc947a2cc17c3e17cd P c9a7b014b4a643818eb8f3cbf44b69e8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.400622 25352 tablet_replica.cc:330] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8: stopping tablet replica
I20240502 08:05:21.400990 25352 raft_consensus.cc:2229] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.401654 25352 raft_consensus.cc:2258] T 478eeaef42cf494eb349a9ca2a4d7b17 P c9a7b014b4a643818eb8f3cbf44b69e8 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 08:05:21.410379 25599 consensus_peers.cc:485] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b -> Peer c9a7b014b4a643818eb8f3cbf44b69e8 (127.24.194.2:39813): Couldn't send request to peer c9a7b014b4a643818eb8f3cbf44b69e8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:21.431066 25352 tablet_server.cc:194] TabletServer@127.24.194.2:0 shutdown complete.
I20240502 08:05:21.448674 25352 tablet_server.cc:178] TabletServer@127.24.194.3:0 shutting down...
I20240502 08:05:21.471621 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:21.472231 25352 tablet_replica.cc:330] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.472846 25352 raft_consensus.cc:2229] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.473665 25352 raft_consensus.cc:2258] T 9f86f3a4a7944c498b4b6bbbb5d606cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.475328 25352 tablet_replica.cc:330] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.475797 25352 raft_consensus.cc:2229] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.476505 25352 raft_consensus.cc:2258] T d8f3937efc374225b60593697f9d6f79 P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.478103 25352 tablet_replica.cc:330] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.478565 25352 raft_consensus.cc:2229] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.478931 25352 raft_consensus.cc:2258] T 010bb756381048b98401bce75b15f03b P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.480383 25352 tablet_replica.cc:330] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.480834 25352 raft_consensus.cc:2229] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.481285 25352 raft_consensus.cc:2258] T f2a7530ae84444f28528e2ecc8d1ed96 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.482667 25352 tablet_replica.cc:330] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.483104 25352 raft_consensus.cc:2229] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.483510 25352 raft_consensus.cc:2258] T 1c877e979fbd43fc947a2cc17c3e17cd P f03e6bea2e5c4e91a4779abc8a57d01b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.484994 25352 tablet_replica.cc:330] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.485394 25352 raft_consensus.cc:2229] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.486121 25352 raft_consensus.cc:2258] T 9bd0117fac064eaea657db7a0263b0a2 P f03e6bea2e5c4e91a4779abc8a57d01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.487550 25352 tablet_replica.cc:330] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.488010 25352 raft_consensus.cc:2229] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.488416 25352 raft_consensus.cc:2258] T 478eeaef42cf494eb349a9ca2a4d7b17 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.489856 25352 tablet_replica.cc:330] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.490278 25352 raft_consensus.cc:2229] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.490669 25352 raft_consensus.cc:2258] T 62009aa71d334e60a3ab5c131817fc2e P f03e6bea2e5c4e91a4779abc8a57d01b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.492084 25352 tablet_replica.cc:330] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b: stopping tablet replica
I20240502 08:05:21.492478 25352 raft_consensus.cc:2229] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:21.492854 25352 raft_consensus.cc:2258] T 0933e210c5b442f38d9598dcc8cb87b1 P f03e6bea2e5c4e91a4779abc8a57d01b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.518783 25352 tablet_server.cc:194] TabletServer@127.24.194.3:0 shutdown complete.
I20240502 08:05:21.536028 25352 master.cc:537] Master@127.24.194.62:36531 shutting down...
I20240502 08:05:21.728948 25352 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:05:21.729563 25352 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:21.729912 25352 tablet_replica.cc:330] T 00000000000000000000000000000000 P 596753f1d4ec462ea98c4a953b0266ad: stopping tablet replica
I20240502 08:05:21.748203 25352 master.cc:552] Master@127.24.194.62:36531 shutdown complete.
c9a7b014b4a643818eb8f3cbf44b69e8 4
e14b3f66048a41d5bfe92033b1925327 4
f03e6bea2e5c4e91a4779abc8a57d01b 1
c9a7b014b4a643818eb8f3cbf44b69e8 3
e14b3f66048a41d5bfe92033b1925327 3
f03e6bea2e5c4e91a4779abc8a57d01b 3
c9a7b014b4a643818eb8f3cbf44b69e8 8
e14b3f66048a41d5bfe92033b1925327 0
f03e6bea2e5c4e91a4779abc8a57d01b 1
c9a7b014b4a643818eb8f3cbf44b69e8 3
e14b3f66048a41d5bfe92033b1925327 3
f03e6bea2e5c4e91a4779abc8a57d01b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30846 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240502 08:05:21.787830 25352 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.194.62:44307
I20240502 08:05:21.788911 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:21.794795 25882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:21.795361 25883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:21.795936 25885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:21.796504 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:21.797343 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:21.797544 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:21.797681 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637121797659 us; error 0 us; skew 500 ppm
I20240502 08:05:21.798226 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:21.800519 25352 webserver.cc:460] Webserver started at http://127.24.194.62:38011/ using document root <none> and password file <none>
I20240502 08:05:21.800976 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:21.801160 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:21.801409 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:21.802587 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/master-0-root/instance:
uuid: "2883e98f465a4cb88f46da43c40ab55b"
format_stamp: "Formatted at 2024-05-02 08:05:21 on dist-test-slave-753s"
I20240502 08:05:21.806702 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 08:05:21.809674 25890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:21.810498 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:21.810721 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/master-0-root
uuid: "2883e98f465a4cb88f46da43c40ab55b"
format_stamp: "Formatted at 2024-05-02 08:05:21 on dist-test-slave-753s"
I20240502 08:05:21.810972 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:21.831445 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:21.832604 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:21.868454 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.62:44307
I20240502 08:05:21.868546 25941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.62:44307 every 8 connection(s)
I20240502 08:05:21.872340 25942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:21.881206 25942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b: Bootstrap starting.
I20240502 08:05:21.885224 25942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:21.889032 25942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b: No bootstrap required, opened a new log
I20240502 08:05:21.890888 25942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER }
I20240502 08:05:21.891328 25942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:21.891505 25942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2883e98f465a4cb88f46da43c40ab55b, State: Initialized, Role: FOLLOWER
I20240502 08:05:21.891980 25942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER }
I20240502 08:05:21.892408 25942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:21.892590 25942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:21.892807 25942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:21.896296 25942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER }
I20240502 08:05:21.896728 25942 leader_election.cc:302] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2883e98f465a4cb88f46da43c40ab55b; no voters:
I20240502 08:05:21.897751 25942 leader_election.cc:288] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:21.898159 25945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:21.899415 25945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 1 LEADER]: Becoming Leader. State: Replica: 2883e98f465a4cb88f46da43c40ab55b, State: Running, Role: LEADER
I20240502 08:05:21.899987 25945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER }
I20240502 08:05:21.900564 25942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:05:21.904992 25946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2883e98f465a4cb88f46da43c40ab55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER } }
I20240502 08:05:21.905258 25947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2883e98f465a4cb88f46da43c40ab55b. Latest consensus state: current_term: 1 leader_uuid: "2883e98f465a4cb88f46da43c40ab55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2883e98f465a4cb88f46da43c40ab55b" member_type: VOTER } }
I20240502 08:05:21.905798 25946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:21.906034 25947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:21.908564 25952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:05:21.913497 25952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:05:21.917250 25352 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:05:21.921808 25952 catalog_manager.cc:1347] Generated new cluster ID: e266ee4cca414f7ea74841f639380d08
I20240502 08:05:21.922144 25952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:05:21.932466 25952 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:05:21.933717 25952 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:05:21.945983 25952 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b: Generated new TSK 0
I20240502 08:05:21.946594 25952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:05:21.950989 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:21.957088 25963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:21.957937 25964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:21.958878 25966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:21.960074 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:21.960901 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:21.961105 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:21.961278 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637121961263 us; error 0 us; skew 500 ppm
I20240502 08:05:21.961843 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:21.964443 25352 webserver.cc:460] Webserver started at http://127.24.194.1:45975/ using document root <none> and password file <none>
I20240502 08:05:21.964938 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:21.965121 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:21.965361 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:21.966480 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-0-root/instance:
uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
format_stamp: "Formatted at 2024-05-02 08:05:21 on dist-test-slave-753s"
I20240502 08:05:21.970800 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:05:21.973858 25971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:21.974594 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:21.974861 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
format_stamp: "Formatted at 2024-05-02 08:05:21 on dist-test-slave-753s"
I20240502 08:05:21.975152 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:21.986696 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:21.987804 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:21.989152 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:21.991297 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:21.991484 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:21.991703 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:21.991866 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.031589 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.1:37523
I20240502 08:05:22.031687 26033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.1:37523 every 8 connection(s)
I20240502 08:05:22.035856 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:22.043896 26038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.045140 26039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.047222 26041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:22.048477 26034 heartbeater.cc:347] Connected to a master server at 127.24.194.62:44307
I20240502 08:05:22.048771 26034 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:22.049186 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:22.049495 26034 heartbeater.cc:511] Master 127.24.194.62:44307 requested a full tablet report, sending...
I20240502 08:05:22.050196 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:22.050400 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:22.050588 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637122050568 us; error 0 us; skew 500 ppm
I20240502 08:05:22.051218 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:22.051568 25907 ts_manager.cc:194] Registered new tserver with Master: f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523)
I20240502 08:05:22.053352 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49396
I20240502 08:05:22.053989 25352 webserver.cc:460] Webserver started at http://127.24.194.2:42317/ using document root <none> and password file <none>
I20240502 08:05:22.054522 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:22.054704 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:22.054955 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:22.056051 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-1-root/instance:
uuid: "4e83328baa9a4c79891c971e0cbd6e44"
format_stamp: "Formatted at 2024-05-02 08:05:22 on dist-test-slave-753s"
I20240502 08:05:22.060415 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:05:22.063592 26046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.064291 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:22.064555 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
uuid: "4e83328baa9a4c79891c971e0cbd6e44"
format_stamp: "Formatted at 2024-05-02 08:05:22 on dist-test-slave-753s"
I20240502 08:05:22.064828 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:22.088356 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:22.089541 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:22.090974 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:22.093055 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:22.093242 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.093463 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:22.093614 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.132026 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.2:46235
I20240502 08:05:22.132134 26108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.2:46235 every 8 connection(s)
I20240502 08:05:22.136451 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:22.143313 26112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.146323 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:22.150231 26109 heartbeater.cc:347] Connected to a master server at 127.24.194.62:44307
I20240502 08:05:22.150707 26109 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:22.151350 26109 heartbeater.cc:511] Master 127.24.194.62:44307 requested a full tablet report, sending...
W20240502 08:05:22.151420 26115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:22.152051 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:22.152876 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:22.153118 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:22.153285 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637122153267 us; error 0 us; skew 500 ppm
I20240502 08:05:22.153599 25907 ts_manager.cc:194] Registered new tserver with Master: 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:22.153812 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:22.155200 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49398
I20240502 08:05:22.157371 25352 webserver.cc:460] Webserver started at http://127.24.194.3:44225/ using document root <none> and password file <none>
I20240502 08:05:22.157871 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:22.158077 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:22.158324 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:22.159437 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-2-root/instance:
uuid: "95176e9e43774b968768d071b45fa063"
format_stamp: "Formatted at 2024-05-02 08:05:22 on dist-test-slave-753s"
I20240502 08:05:22.163743 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:05:22.166925 26120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.167634 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:22.167903 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
uuid: "95176e9e43774b968768d071b45fa063"
format_stamp: "Formatted at 2024-05-02 08:05:22 on dist-test-slave-753s"
I20240502 08:05:22.168169 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714637090785390-25352-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:22.181412 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:22.182663 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:22.184068 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:22.186193 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:22.186369 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.186599 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:22.186736 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.225212 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.3:36727
I20240502 08:05:22.225317 26182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.3:36727 every 8 connection(s)
I20240502 08:05:22.237700 26183 heartbeater.cc:347] Connected to a master server at 127.24.194.62:44307
I20240502 08:05:22.238106 26183 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:22.238790 26183 heartbeater.cc:511] Master 127.24.194.62:44307 requested a full tablet report, sending...
I20240502 08:05:22.240700 25907 ts_manager.cc:194] Registered new tserver with Master: 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:22.241302 25352 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013170498s
I20240502 08:05:22.241973 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49402
I20240502 08:05:22.243816 25352 test_util.cc:268] Using random seed: 278334677
I20240502 08:05:22.264849 25907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49418:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:05:22.267706 25907 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:22.343469 26148 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0ff0d20ea4e319df305a9712534d5 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 08:05:22.345064 26148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0ff0d20ea4e319df305a9712534d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.346309 26145 tablet_service.cc:1467] Processing CreateTablet for tablet 0834e718719042dfa1caebe291dfafcd (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 08:05:22.347682 26145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834e718719042dfa1caebe291dfafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.347435 26147 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2ac7935c48411995c00d1e411b5caf (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 08:05:22.348798 26147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2ac7935c48411995c00d1e411b5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.351369 26146 tablet_service.cc:1467] Processing CreateTablet for tablet eee0db9a1d91441d85d161036d046a8a (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 08:05:22.351779 26144 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0aa5c067f48b2a1cd8bdd734c914c (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 08:05:22.352818 26146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee0db9a1d91441d85d161036d046a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.352530 25998 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2ac7935c48411995c00d1e411b5caf (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 08:05:22.353871 25998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2ac7935c48411995c00d1e411b5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.354152 26143 tablet_service.cc:1467] Processing CreateTablet for tablet 952f1cd3bd434b31ac693cc46c0eaaed (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 08:05:22.354876 26142 tablet_service.cc:1467] Processing CreateTablet for tablet bd5e0423799c4a568e0ab3ba7bd5c594 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 08:05:22.355288 26144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0aa5c067f48b2a1cd8bdd734c914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.360690 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9767b330e7404383f087c7d44bd119 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 08:05:22.355232 25997 tablet_service.cc:1467] Processing CreateTablet for tablet eee0db9a1d91441d85d161036d046a8a (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 08:05:22.356658 26072 tablet_service.cc:1467] Processing CreateTablet for tablet eee0db9a1d91441d85d161036d046a8a (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 08:05:22.361573 26068 tablet_service.cc:1467] Processing CreateTablet for tablet bd5e0423799c4a568e0ab3ba7bd5c594 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 08:05:22.362028 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9767b330e7404383f087c7d44bd119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.362833 26072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee0db9a1d91441d85d161036d046a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.363584 26066 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9767b330e7404383f087c7d44bd119 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 08:05:22.356916 26141 tablet_service.cc:1467] Processing CreateTablet for tablet 0232ea64cb9a46c38908f3405e4625cb (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 08:05:22.365706 26141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0232ea64cb9a46c38908f3405e4625cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.365782 26065 tablet_service.cc:1467] Processing CreateTablet for tablet bf058a946c04431eb4146726bbf044d7 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 08:05:22.366761 25997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee0db9a1d91441d85d161036d046a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.367918 25994 tablet_service.cc:1467] Processing CreateTablet for tablet 952f1cd3bd434b31ac693cc46c0eaaed (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 08:05:22.357017 25996 tablet_service.cc:1467] Processing CreateTablet for tablet 0834e718719042dfa1caebe291dfafcd (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 08:05:22.368643 25991 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9767b330e7404383f087c7d44bd119 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 08:05:22.369266 25994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f1cd3bd434b31ac693cc46c0eaaed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.358806 26139 tablet_service.cc:1467] Processing CreateTablet for tablet bf058a946c04431eb4146726bbf044d7 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 08:05:22.358919 26070 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0aa5c067f48b2a1cd8bdd734c914c (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 08:05:22.359019 26073 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2ac7935c48411995c00d1e411b5caf (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 08:05:22.358526 26071 tablet_service.cc:1467] Processing CreateTablet for tablet 0834e718719042dfa1caebe291dfafcd (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 08:05:22.375319 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9767b330e7404383f087c7d44bd119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.375174 26069 tablet_service.cc:1467] Processing CreateTablet for tablet 952f1cd3bd434b31ac693cc46c0eaaed (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 08:05:22.375557 25995 tablet_service.cc:1467] Processing CreateTablet for tablet c9f0aa5c067f48b2a1cd8bdd734c914c (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 08:05:22.376327 26073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2ac7935c48411995c00d1e411b5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.351369 25999 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0ff0d20ea4e319df305a9712534d5 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 08:05:22.376715 26142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5e0423799c4a568e0ab3ba7bd5c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.377192 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f1cd3bd434b31ac693cc46c0eaaed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.377039 25990 tablet_service.cc:1467] Processing CreateTablet for tablet bf058a946c04431eb4146726bbf044d7 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 08:05:22.378055 26071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834e718719042dfa1caebe291dfafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.378376 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf058a946c04431eb4146726bbf044d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.379032 26074 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0ff0d20ea4e319df305a9712534d5 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 08:05:22.379937 25993 tablet_service.cc:1467] Processing CreateTablet for tablet bd5e0423799c4a568e0ab3ba7bd5c594 (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 08:05:22.371392 25992 tablet_service.cc:1467] Processing CreateTablet for tablet 0232ea64cb9a46c38908f3405e4625cb (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 08:05:22.371374 26067 tablet_service.cc:1467] Processing CreateTablet for tablet 0232ea64cb9a46c38908f3405e4625cb (DEFAULT_TABLE table=test-workload [id=e7023dbfa18c44c5b5f36d8e6d2ce96e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 08:05:22.381258 26068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5e0423799c4a568e0ab3ba7bd5c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.381973 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0232ea64cb9a46c38908f3405e4625cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.382653 26070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0aa5c067f48b2a1cd8bdd734c914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.383436 25991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9767b330e7404383f087c7d44bd119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.386359 26065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf058a946c04431eb4146726bbf044d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.386581 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834e718719042dfa1caebe291dfafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.390578 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5e0423799c4a568e0ab3ba7bd5c594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.394577 25990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf058a946c04431eb4146726bbf044d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.398620 25992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0232ea64cb9a46c38908f3405e4625cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.400880 25995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f0aa5c067f48b2a1cd8bdd734c914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.402388 25999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0ff0d20ea4e319df305a9712534d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.403793 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0ff0d20ea4e319df305a9712534d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.404976 26143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f1cd3bd434b31ac693cc46c0eaaed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.485833 26198 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.536111 26199 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.539203 26198 tablet_bootstrap.cc:654] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.545688 26199 tablet_bootstrap.cc:654] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.558594 26200 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.580176 26198 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.580691 26198 ts_tablet_manager.cc:1367] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.011s
I20240502 08:05:22.587411 26198 raft_consensus.cc:348] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.588166 26198 raft_consensus.cc:374] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.588449 26198 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.589094 26198 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.592141 26200 tablet_bootstrap.cc:654] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.597546 26034 heartbeater.cc:503] Master 127.24.194.62:44307 was elected leader, sending a full tablet report...
I20240502 08:05:22.598094 26198 ts_tablet_manager.cc:1398] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20240502 08:05:22.599023 26198 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.600962 26199 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.601523 26199 ts_tablet_manager.cc:1367] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.019s
I20240502 08:05:22.605600 26199 raft_consensus.cc:348] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.606688 26199 raft_consensus.cc:374] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.607075 26199 raft_consensus.cc:729] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.607329 26198 tablet_bootstrap.cc:654] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.607973 26199 consensus_queue.cc:260] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.613165 26200 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.613551 26200 ts_tablet_manager.cc:1367] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.010s
I20240502 08:05:22.615504 26200 raft_consensus.cc:348] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.616024 26200 raft_consensus.cc:374] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.616284 26200 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.616873 26200 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.619822 26200 ts_tablet_manager.cc:1398] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 08:05:22.620126 26198 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.620770 26198 ts_tablet_manager.cc:1367] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20240502 08:05:22.624294 26198 raft_consensus.cc:348] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.625524 26198 raft_consensus.cc:374] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.625557 26183 heartbeater.cc:503] Master 127.24.194.62:44307 was elected leader, sending a full tablet report...
I20240502 08:05:22.626086 26198 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.626811 26198 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.629290 26198 ts_tablet_manager.cc:1398] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:05:22.630218 26198 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.636111 26198 tablet_bootstrap.cc:654] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.638149 26109 heartbeater.cc:503] Master 127.24.194.62:44307 was elected leader, sending a full tablet report...
I20240502 08:05:22.638671 26199 ts_tablet_manager.cc:1398] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.037s user 0.016s sys 0.018s
I20240502 08:05:22.639529 26199 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.642573 26200 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.645413 26199 tablet_bootstrap.cc:654] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.648427 26198 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.649012 26198 ts_tablet_manager.cc:1367] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240502 08:05:22.651868 26199 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.652366 26199 ts_tablet_manager.cc:1367] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 08:05:22.652696 26200 tablet_bootstrap.cc:654] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.653952 26198 raft_consensus.cc:348] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.654740 26198 raft_consensus.cc:374] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.655061 26198 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.654950 26199 raft_consensus.cc:348] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.655855 26199 raft_consensus.cc:374] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.656178 26199 raft_consensus.cc:729] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.656229 26198 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.657003 26199 consensus_queue.cc:260] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.659211 26199 ts_tablet_manager.cc:1398] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:05:22.659606 26198 ts_tablet_manager.cc:1398] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:05:22.660071 26199 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.660836 26198 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.668917 26199 tablet_bootstrap.cc:654] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.669168 26198 tablet_bootstrap.cc:654] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.679466 26207 raft_consensus.cc:482] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.680213 26199 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.679948 26207 raft_consensus.cc:504] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.680667 26199 ts_tablet_manager.cc:1367] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.007s
I20240502 08:05:22.682507 26207 leader_election.cc:288] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523)
I20240502 08:05:22.685390 26199 raft_consensus.cc:348] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.686184 26199 raft_consensus.cc:374] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.686472 26199 raft_consensus.cc:729] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.687260 26199 consensus_queue.cc:260] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.687304 26198 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.689217 26198 ts_tablet_manager.cc:1367] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.004s
I20240502 08:05:22.689560 26199 ts_tablet_manager.cc:1398] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 08:05:22.690527 26199 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.692680 26198 raft_consensus.cc:348] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.693667 26198 raft_consensus.cc:374] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.694064 26198 raft_consensus.cc:729] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.695178 26198 consensus_queue.cc:260] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.696219 26199 tablet_bootstrap.cc:654] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.703872 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:22.704658 26198 ts_tablet_manager.cc:1398] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20240502 08:05:22.705035 26009 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:22.707099 26205 raft_consensus.cc:482] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.707567 26205 raft_consensus.cc:504] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.708617 26124 leader_election.cc:302] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:22.708911 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:22.709533 26207 raft_consensus.cc:2781] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:22.709776 26205 leader_election.cc:288] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:22.710604 26199 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
W20240502 08:05:22.710736 26122 leader_election.cc:341] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.711126 26199 ts_tablet_manager.cc:1367] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20240502 08:05:22.709914 26207 raft_consensus.cc:482] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:22.711969 26207 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.713897 26199 raft_consensus.cc:348] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.714684 26199 raft_consensus.cc:374] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.714977 26199 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.715651 26199 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.717847 26199 ts_tablet_manager.cc:1398] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:22.718744 26199 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.721557 26200 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.722188 26200 ts_tablet_manager.cc:1367] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.080s user 0.020s sys 0.021s
I20240502 08:05:22.723187 26207 raft_consensus.cc:504] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.724654 26199 tablet_bootstrap.cc:654] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.725510 26200 raft_consensus.cc:348] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.727468 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:22.728044 26207 leader_election.cc:288] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523)
W20240502 08:05:22.728770 26122 leader_election.cc:341] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.728706 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:22.726264 26200 raft_consensus.cc:374] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.729365 26009 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.729622 26200 raft_consensus.cc:729] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.730481 26200 consensus_queue.cc:260] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.732558 26200 ts_tablet_manager.cc:1398] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:05:22.737133 26009 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:22.738268 26124 leader_election.cc:302] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters: 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:22.739140 26207 raft_consensus.cc:2781] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.739687 26207 raft_consensus.cc:686] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:22.740720 26207 consensus_queue.cc:237] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:22.745406 26200 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.750716 26198 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.754244 26200 tablet_bootstrap.cc:654] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.754325 26205 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.754855 26205 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.757869 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:22.758603 26008 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
I20240502 08:05:22.759824 26050 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:22.760774 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:22.759845 26205 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:22.761164 25906 catalog_manager.cc:5484] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:22.762158 26205 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:22.762524 26205 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:22.762868 26205 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.778461 26200 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.778985 26200 ts_tablet_manager.cc:1367] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.002s
I20240502 08:05:22.781651 26200 raft_consensus.cc:348] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.782531 26198 tablet_bootstrap.cc:654] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.782459 26200 raft_consensus.cc:374] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.782912 26200 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.783838 26200 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.788864 26205 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.785981 26200 ts_tablet_manager.cc:1398] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 08:05:22.792897 26200 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.792707 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
W20240502 08:05:22.794680 26048 leader_election.cc:341] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.795465 26205 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:22.796784 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:22.798115 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:22.798751 26009 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:22.798902 26048 leader_election.cc:341] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.798022 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
W20240502 08:05:22.801448 26048 leader_election.cc:341] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.801764 26199 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.802124 26200 tablet_bootstrap.cc:654] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.801208 26198 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.802299 26199 ts_tablet_manager.cc:1367] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.084s user 0.056s sys 0.019s
I20240502 08:05:22.802857 26198 ts_tablet_manager.cc:1367] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.005s
I20240502 08:05:22.805188 26199 raft_consensus.cc:348] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.805941 26199 raft_consensus.cc:374] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.805478 26198 raft_consensus.cc:348] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.806453 26198 raft_consensus.cc:374] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.806847 26009 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:22.806854 26198 raft_consensus.cc:729] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.807288 26199 raft_consensus.cc:729] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
W20240502 08:05:22.808449 26050 leader_election.cc:341] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.808085 26199 consensus_queue.cc:260] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.810534 26199 ts_tablet_manager.cc:1398] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:05:22.809033 26050 leader_election.cc:302] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44; no voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:22.811362 26220 raft_consensus.cc:2726] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:22.811555 26199 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.808125 26198 consensus_queue.cc:260] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.812865 26050 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters: 95176e9e43774b968768d071b45fa063
I20240502 08:05:22.813656 26220 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.814142 26220 raft_consensus.cc:686] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:22.815026 26220 consensus_queue.cc:237] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.817828 26199 tablet_bootstrap.cc:654] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.817871 26198 ts_tablet_manager.cc:1398] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240502 08:05:22.822521 26198 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.826740 26200 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.827229 26200 ts_tablet_manager.cc:1367] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.003s
I20240502 08:05:22.828161 26198 tablet_bootstrap.cc:654] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.829941 26200 raft_consensus.cc:348] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.830718 26200 raft_consensus.cc:374] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.831203 26200 raft_consensus.cc:729] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.831980 26200 consensus_queue.cc:260] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.835407 26200 ts_tablet_manager.cc:1398] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:05:22.836342 26200 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.837703 25906 catalog_manager.cc:5484] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:22.839948 26203 raft_consensus.cc:482] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.840356 26203 raft_consensus.cc:504] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.841805 26199 tablet_bootstrap.cc:492] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.842784 26199 ts_tablet_manager.cc:1367] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.013s
I20240502 08:05:22.842924 26203 leader_election.cc:288] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:22.845664 26199 raft_consensus.cc:348] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.846680 26199 raft_consensus.cc:374] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.847060 26199 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.848588 26198 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.848114 26199 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.849076 26198 ts_tablet_manager.cc:1367] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20240502 08:05:22.851066 26199 ts_tablet_manager.cc:1398] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:05:22.851773 26198 raft_consensus.cc:348] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.852259 26199 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.852964 26200 tablet_bootstrap.cc:654] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.854454 26218 raft_consensus.cc:482] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.854921 26218 raft_consensus.cc:504] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.857245 26218 leader_election.cc:288] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:22.858016 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:22.858682 26009 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:22.858708 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:22.860258 26199 tablet_bootstrap.cc:654] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.860643 26124 leader_election.cc:302] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
W20240502 08:05:22.861878 26122 leader_election.cc:341] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:22.862728 26218 raft_consensus.cc:2781] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:22.863101 26218 raft_consensus.cc:482] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:22.863431 26218 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.852504 26198 raft_consensus.cc:374] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.865028 26198 raft_consensus.cc:729] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.865770 26198 consensus_queue.cc:260] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.867996 26198 ts_tablet_manager.cc:1398] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 08:05:22.870646 26198 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.870684 26218 raft_consensus.cc:504] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.873430 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:22.874082 26009 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.876770 26198 tablet_bootstrap.cc:654] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.882501 26009 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:22.883612 26124 leader_election.cc:302] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:22.884444 26207 raft_consensus.cc:2781] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.888778 26200 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.889286 26200 ts_tablet_manager.cc:1367] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.005s
I20240502 08:05:22.890656 26199 tablet_bootstrap.cc:492] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.891283 26199 ts_tablet_manager.cc:1367] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.025s
I20240502 08:05:22.892035 26200 raft_consensus.cc:348] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.892778 26200 raft_consensus.cc:374] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.893103 26200 raft_consensus.cc:729] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.893823 26200 consensus_queue.cc:260] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.899190 26218 leader_election.cc:288] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:22.898990 26199 raft_consensus.cc:348] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.899768 26199 raft_consensus.cc:374] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.899855 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:22.900100 26199 raft_consensus.cc:729] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.902660 26199 consensus_queue.cc:260] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.903642 26084 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.904047 26198 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.904501 26198 ts_tablet_manager.cc:1367] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.002s
I20240502 08:05:22.905042 26199 ts_tablet_manager.cc:1398] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 08:05:22.906837 26207 raft_consensus.cc:686] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:22.907191 26198 raft_consensus.cc:348] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.907960 26198 raft_consensus.cc:374] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.907703 26207 consensus_queue.cc:237] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.908283 26198 raft_consensus.cc:729] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.908983 26198 consensus_queue.cc:260] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.910565 26084 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:22.911720 26198 ts_tablet_manager.cc:1398] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20240502 08:05:22.905925 26199 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.912777 26198 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
W20240502 08:05:22.913568 25955 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:22.917323 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:22.919052 26198 tablet_bootstrap.cc:654] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.919173 26199 tablet_bootstrap.cc:654] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.921700 26200 ts_tablet_manager.cc:1398] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.032s user 0.011s sys 0.000s
I20240502 08:05:22.922992 26200 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.928714 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:22.930717 26200 tablet_bootstrap.cc:654] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.930383 25906 catalog_manager.cc:5484] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:22.931491 25973 leader_election.cc:302] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c2fbdd0db749cf8cf558cac1b77e73; no voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063
I20240502 08:05:22.932511 26203 raft_consensus.cc:2726] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:22.938192 26199 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.938701 26199 ts_tablet_manager.cc:1367] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.000s
I20240502 08:05:22.939786 26198 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.940310 26198 ts_tablet_manager.cc:1367] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.004s
I20240502 08:05:22.941411 26199 raft_consensus.cc:348] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.942276 26199 raft_consensus.cc:374] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.942551 26199 raft_consensus.cc:729] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.943681 26200 tablet_bootstrap.cc:492] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.943298 26199 consensus_queue.cc:260] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.944190 26200 ts_tablet_manager.cc:1367] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240502 08:05:22.943289 26198 raft_consensus.cc:348] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.944856 26198 raft_consensus.cc:374] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.945132 26198 raft_consensus.cc:729] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.945720 26198 consensus_queue.cc:260] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.947678 26198 ts_tablet_manager.cc:1398] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:05:22.948738 26199 ts_tablet_manager.cc:1398] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:05:22.949656 26199 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.949656 26198 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.948678 26200 raft_consensus.cc:348] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.951154 26200 raft_consensus.cc:374] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.951449 26200 raft_consensus.cc:729] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.952234 26200 consensus_queue.cc:260] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.954668 26200 ts_tablet_manager.cc:1398] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20240502 08:05:22.955085 26199 tablet_bootstrap.cc:654] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.955575 26200 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.955097 26198 tablet_bootstrap.cc:654] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.960146 26199 tablet_bootstrap.cc:492] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:22.960795 26199 ts_tablet_manager.cc:1367] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:05:22.961699 26200 tablet_bootstrap.cc:654] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.964334 26199 raft_consensus.cc:348] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.965459 26199 raft_consensus.cc:374] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.965852 26199 raft_consensus.cc:729] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.966660 26199 consensus_queue.cc:260] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.968930 26199 ts_tablet_manager.cc:1398] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:05:22.969868 26199 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: Bootstrap starting.
I20240502 08:05:22.969883 26198 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:22.970561 26198 ts_tablet_manager.cc:1367] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20240502 08:05:22.971487 26200 tablet_bootstrap.cc:492] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:22.971941 26200 ts_tablet_manager.cc:1367] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.001s
I20240502 08:05:22.972895 26198 raft_consensus.cc:348] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.973627 26198 raft_consensus.cc:374] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.973910 26198 raft_consensus.cc:729] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.974655 26198 consensus_queue.cc:260] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.974678 26200 raft_consensus.cc:348] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.975561 26200 raft_consensus.cc:374] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.975998 26200 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.976197 26199 tablet_bootstrap.cc:654] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.976874 26200 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.979508 26200 ts_tablet_manager.cc:1398] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:05:22.980434 26200 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:22.982077 26207 raft_consensus.cc:482] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:22.982555 26207 raft_consensus.cc:504] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:22.982930 26198 ts_tablet_manager.cc:1398] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 08:05:22.984544 26207 leader_election.cc:288] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:22.986066 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:22.986708 26009 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:22.987844 26124 leader_election.cc:302] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:22.988225 26198 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: Bootstrap starting.
I20240502 08:05:22.989580 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:22.990257 26084 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:22.991840 26207 raft_consensus.cc:2781] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:22.992209 26207 raft_consensus.cc:482] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:22.992559 26207 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.995010 26198 tablet_bootstrap.cc:654] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:23.002660 26199 tablet_bootstrap.cc:492] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: No bootstrap required, opened a new log
I20240502 08:05:23.003342 26199 ts_tablet_manager.cc:1367] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.007s
I20240502 08:05:23.005468 26200 tablet_bootstrap.cc:654] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:23.006251 26199 raft_consensus.cc:348] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:23.006907 26199 raft_consensus.cc:374] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.007171 26199 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.007513 26207 raft_consensus.cc:504] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.007752 26198 tablet_bootstrap.cc:492] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: No bootstrap required, opened a new log
I20240502 08:05:23.007791 26199 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:23.008363 26198 ts_tablet_manager.cc:1367] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.005s
I20240502 08:05:23.009821 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.010428 26009 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.010571 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.011193 26084 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.010929 26198 raft_consensus.cc:348] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.011791 26198 raft_consensus.cc:374] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.012069 26198 raft_consensus.cc:729] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.012832 26198 consensus_queue.cc:260] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.017239 26198 ts_tablet_manager.cc:1398] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 08:05:23.017145 26084 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:23.018256 26199 ts_tablet_manager.cc:1398] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 08:05:23.018203 26122 leader_election.cc:302] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.019274 26233 raft_consensus.cc:2781] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.019760 26233 raft_consensus.cc:686] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:23.020576 26233 consensus_queue.cc:237] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.018198 26207 leader_election.cc:288] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:23.024858 26203 raft_consensus.cc:482] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.025312 26203 raft_consensus.cc:504] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.027238 26203 leader_election.cc:288] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.027863 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:23.028174 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:23.028512 26084 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 0.
I20240502 08:05:23.028863 26200 tablet_bootstrap.cc:492] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:23.028745 26158 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 0.
I20240502 08:05:23.028684 26009 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:23.029421 26200 ts_tablet_manager.cc:1367] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.005s
I20240502 08:05:23.029755 25973 leader_election.cc:302] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.032052 26203 raft_consensus.cc:2781] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.032179 26200 raft_consensus.cc:348] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.032457 26203 raft_consensus.cc:482] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.032763 26200 raft_consensus.cc:374] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.032845 26203 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.033161 26200 raft_consensus.cc:729] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.033738 25907 catalog_manager.cc:5484] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:23.034250 26200 consensus_queue.cc:260] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.036607 26200 ts_tablet_manager.cc:1398] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 08:05:23.037303 26200 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:23.040302 26220 raft_consensus.cc:482] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.040751 26220 raft_consensus.cc:504] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.042459 26220 leader_election.cc:288] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.043736 26200 tablet_bootstrap.cc:654] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:23.044751 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:23.045444 26158 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
I20240502 08:05:23.046767 26048 leader_election.cc:302] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.047588 26220 raft_consensus.cc:2781] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.047959 26220 raft_consensus.cc:482] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.048310 26220 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.048208 26203 raft_consensus.cc:504] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.047971 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:23.051195 26009 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
I20240502 08:05:23.052268 26203 leader_election.cc:288] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.053225 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.053654 26084 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.053690 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.054344 26158 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.055588 26200 tablet_bootstrap.cc:492] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:23.056123 26200 ts_tablet_manager.cc:1367] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20240502 08:05:23.058729 26200 raft_consensus.cc:348] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.059478 26200 raft_consensus.cc:374] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.059796 26200 raft_consensus.cc:729] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.060705 26084 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 1.
I20240502 08:05:23.060565 26200 consensus_queue.cc:260] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.061661 26220 raft_consensus.cc:504] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.061929 25973 leader_election.cc:302] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.062840 26203 raft_consensus.cc:2781] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.062948 26200 ts_tablet_manager.cc:1398] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:23.063774 26220 leader_election.cc:288] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.063967 26200 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: Bootstrap starting.
I20240502 08:05:23.065312 26158 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 1.
I20240502 08:05:23.065454 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.066090 26157 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.065959 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.066557 26220 raft_consensus.cc:482] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.066848 26009 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.066764 26205 raft_consensus.cc:482] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.069487 26220 raft_consensus.cc:504] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.069801 26205 raft_consensus.cc:504] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.073194 26205 leader_election.cc:288] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.075516 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:23.076296 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:23.076946 26008 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
I20240502 08:05:23.076936 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:23.075620 26220 leader_election.cc:288] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.077987 26007 raft_consensus.cc:2454] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
W20240502 08:05:23.077982 26048 leader_election.cc:341] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:23.080503 26050 leader_election.cc:302] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.075536 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:23.081435 26050 leader_election.cc:302] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters: 95176e9e43774b968768d071b45fa063
I20240502 08:05:23.081673 26158 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 0.
I20240502 08:05:23.082556 26205 raft_consensus.cc:2781] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.082933 26205 raft_consensus.cc:482] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.083259 26205 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.086458 26009 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.087464 26050 leader_election.cc:302] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.089972 26220 raft_consensus.cc:2781] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.090992 26220 raft_consensus.cc:482] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.092252 26220 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.091212 26157 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.095475 26200 tablet_bootstrap.cc:654] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:23.090281 26203 raft_consensus.cc:686] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.097900 26247 raft_consensus.cc:482] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.097306 26203 consensus_queue.cc:237] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.098531 26247 raft_consensus.cc:504] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.096367 26205 raft_consensus.cc:504] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.103202 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063" is_pre_election: true
I20240502 08:05:23.104003 26157 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 0.
I20240502 08:05:23.104684 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:23.104140 26248 raft_consensus.cc:2781] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.105319 25973 leader_election.cc:302] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.105314 26084 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 0.
I20240502 08:05:23.106746 26203 raft_consensus.cc:2781] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.107129 26203 raft_consensus.cc:482] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.107457 26203 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.108160 26247 leader_election.cc:288] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.109673 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.110294 26009 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.112548 26200 tablet_bootstrap.cc:492] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: No bootstrap required, opened a new log
I20240502 08:05:23.113173 26200 ts_tablet_manager.cc:1367] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.000s
I20240502 08:05:23.113018 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.113687 26157 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.114270 26203 raft_consensus.cc:504] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.116111 26203 leader_election.cc:288] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.115959 26200 raft_consensus.cc:348] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.116847 26200 raft_consensus.cc:374] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.117269 26200 raft_consensus.cc:729] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.118513 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.118222 26200 consensus_queue.cc:260] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.118698 26205 leader_election.cc:288] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.119140 26084 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.121012 26200 ts_tablet_manager.cc:1398] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240502 08:05:23.117559 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.122290 26158 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.123759 26220 raft_consensus.cc:504] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.124382 25906 catalog_manager.cc:5484] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 1 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.126154 26248 raft_consensus.cc:686] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.127753 26157 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.128021 26084 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 1.
I20240502 08:05:23.129276 26048 leader_election.cc:302] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.129419 26220 leader_election.cc:288] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.130106 26220 raft_consensus.cc:2781] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.130764 26220 raft_consensus.cc:686] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.131628 26220 consensus_queue.cc:237] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.132330 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.132926 26007 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.127004 26248 consensus_queue.cc:237] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.136194 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.136813 26157 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.139133 26158 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 1.
I20240502 08:05:23.128038 26009 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.143389 25973 leader_election.cc:302] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.144299 26203 raft_consensus.cc:2781] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.145510 26203 raft_consensus.cc:686] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.146880 26157 raft_consensus.cc:2454] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.146653 26203 consensus_queue.cc:237] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.147961 26048 leader_election.cc:302] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.148993 26251 raft_consensus.cc:2781] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.149478 26251 raft_consensus.cc:686] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.152024 26007 raft_consensus.cc:2454] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 1.
I20240502 08:05:23.155161 26233 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.150369 26251 consensus_queue.cc:237] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.160041 25906 catalog_manager.cc:5484] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.171176 25907 catalog_manager.cc:5484] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 1 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.171934 26207 raft_consensus.cc:482] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.172425 26207 raft_consensus.cc:504] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.174321 26207 leader_election.cc:288] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:23.176610 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:23.176680 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:23.177386 26084 raft_consensus.cc:2454] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:23.177552 26009 raft_consensus.cc:2454] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 0.
I20240502 08:05:23.182492 26124 leader_election.cc:302] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.183583 26233 raft_consensus.cc:2781] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:23.184001 26233 raft_consensus.cc:482] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:23.184607 26233 raft_consensus.cc:3037] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.186197 25907 catalog_manager.cc:5484] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.188117 25907 catalog_manager.cc:5484] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 1 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.189097 26084 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.192932 26233 raft_consensus.cc:504] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.197962 26233 leader_election.cc:288] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:23.198571 26232 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:23.198755 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.198992 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf058a946c04431eb4146726bbf044d7" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.199412 26007 raft_consensus.cc:3037] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.199523 26084 raft_consensus.cc:3037] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.206516 26084 raft_consensus.cc:2454] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:23.209266 26122 leader_election.cc:302] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.210389 26007 raft_consensus.cc:2454] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 1.
I20240502 08:05:23.210642 26232 raft_consensus.cc:2781] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.211210 26232 raft_consensus.cc:686] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:23.212265 26232 consensus_queue.cc:237] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.220361 25907 catalog_manager.cc:5484] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:23.248451 26157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "95176e9e43774b968768d071b45fa063"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:56730
I20240502 08:05:23.249024 26157 raft_consensus.cc:595] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:23.263067 26083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:23.263535 26083 raft_consensus.cc:595] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:23.267882 26083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:23.268352 26083 raft_consensus.cc:595] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:23.272476 26083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:23.272926 26083 raft_consensus.cc:595] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:23.279348 26157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594"
dest_uuid: "95176e9e43774b968768d071b45fa063"
mode: GRACEFUL
new_leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:56730
I20240502 08:05:23.279810 26157 raft_consensus.cc:595] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Received request to transfer leadership to TS 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:23.290757 26007 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:58556
I20240502 08:05:23.291363 26007 raft_consensus.cc:595] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Received request to transfer leadership to TS 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:23.295953 26157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119"
dest_uuid: "95176e9e43774b968768d071b45fa063"
mode: GRACEFUL
new_leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:56730
I20240502 08:05:23.296383 26157 raft_consensus.cc:595] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Received request to transfer leadership to TS 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:23.389269 26232 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:23.389760 26232 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.391383 26232 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:23.392319 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" is_pre_election: true
I20240502 08:05:23.392511 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44" is_pre_election: true
I20240502 08:05:23.392930 26007 raft_consensus.cc:2379] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95176e9e43774b968768d071b45fa063 in current term 1: Already voted for candidate 4e83328baa9a4c79891c971e0cbd6e44 in this term.
I20240502 08:05:23.393797 26124 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95176e9e43774b968768d071b45fa063; no voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:23.394464 26232 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.398957 26232 raft_consensus.cc:2726] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:23.428521 26251 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.434026 26248 raft_consensus.cc:979] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.434506 26220 raft_consensus.cc:1067] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.436046 26009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:23.438831 26220 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.439690 26009 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.440106 26009 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.447142 26009 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.451431 26009 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.451903 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.452530 26083 raft_consensus.cc:3032] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.452430 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.452896 26083 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.453032 26157 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.453665 26083 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.454726 26083 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.458143 26157 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.459090 25973 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.459348 26083 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.459717 26203 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.460145 26203 raft_consensus.cc:686] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.460894 26203 consensus_queue.cc:237] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.467795 25907 catalog_manager.cc:5484] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 1 to 2, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.484975 26232 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.490808 26232 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.499327 26207 consensus_queue.cc:1035] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.503109 26218 raft_consensus.cc:979] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: : Instructing follower 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.503543 26218 raft_consensus.cc:1067] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Signalling peer 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.506466 26084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:53536
I20240502 08:05:23.507057 26084 raft_consensus.cc:482] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.507488 26084 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.513806 26207 consensus_queue.cc:1035] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.515432 26084 raft_consensus.cc:504] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.521406 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.521998 26084 leader_election.cc:288] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.522521 26009 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.524039 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.524695 26157 raft_consensus.cc:3032] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.525049 26157 raft_consensus.cc:729] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:23.525831 26157 consensus_queue.cc:260] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.527149 26157 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.528577 26203 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.533280 26157 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.534210 26220 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.536326 26048 leader_election.cc:302] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.540284 26009 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.540829 26220 raft_consensus.cc:2781] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.544936 26220 raft_consensus.cc:686] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.540611 26246 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:23.552471 26220 consensus_queue.cc:237] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.556843 26207 raft_consensus.cc:979] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: : Instructing follower 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.557336 26232 raft_consensus.cc:1067] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Signalling peer 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.559186 26246 raft_consensus.cc:979] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.559423 26084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:53536
I20240502 08:05:23.559613 26246 raft_consensus.cc:1067] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Signalling peer 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:23.559988 26084 raft_consensus.cc:482] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.560381 26084 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.561586 26083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:53548
I20240502 08:05:23.562187 26083 raft_consensus.cc:482] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.563488 26083 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.567137 26220 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.570309 26084 raft_consensus.cc:504] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.570600 26083 raft_consensus.cc:504] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.572734 26084 leader_election.cc:288] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.574204 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.574805 26007 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.575016 26083 leader_election.cc:288] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.575304 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9767b330e7404383f087c7d44bd119" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.575881 26158 raft_consensus.cc:3032] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.576660 26158 raft_consensus.cc:729] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:23.577414 26158 consensus_queue.cc:260] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.578637 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:23.578874 26158 raft_consensus.cc:3037] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.579314 26008 raft_consensus.cc:3032] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.578774 25907 catalog_manager.cc:5484] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 1 to 2, leader changed from 95176e9e43774b968768d071b45fa063 (127.24.194.3) to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.579639 26008 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.580653 26008 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.584275 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.584900 26157 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.589170 26220 raft_consensus.cc:979] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.589121 26008 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.589713 26220 raft_consensus.cc:1067] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.597757 26158 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.598956 26008 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.598920 26048 leader_election.cc:302] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:23.600029 26248 raft_consensus.cc:2781] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.600121 26050 leader_election.cc:302] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.600313 26009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:23.600570 26248 raft_consensus.cc:686] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.600931 26009 raft_consensus.cc:482] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.601187 26205 raft_consensus.cc:2781] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.601334 26009 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.601703 26205 raft_consensus.cc:686] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.601423 26248 consensus_queue.cc:237] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.602591 26205 consensus_queue.cc:237] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.604552 26157 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.608593 26232 raft_consensus.cc:979] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.609002 26232 raft_consensus.cc:1067] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.609527 26007 raft_consensus.cc:2454] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 2.
I20240502 08:05:23.611898 25906 catalog_manager.cc:5484] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 1 to 2, leader changed from 95176e9e43774b968768d071b45fa063 (127.24.194.3) to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.612579 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58538
I20240502 08:05:23.613481 26008 raft_consensus.cc:482] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.613691 26205 consensus_queue.cc:1035] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.613924 26008 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.613859 26009 raft_consensus.cc:504] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.616632 26009 leader_election.cc:288] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.616564 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.617297 26083 raft_consensus.cc:3032] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.617202 26157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.617650 26083 raft_consensus.cc:729] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.617787 26157 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.618438 26083 consensus_queue.cc:260] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.619634 26083 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.624385 26008 raft_consensus.cc:504] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:23.625095 26157 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.626252 25973 leader_election.cc:302] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.626365 26008 leader_election.cc:288] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Requested vote from peers 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:23.626464 26083 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.628000 26247 raft_consensus.cc:2781] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.628089 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.628511 26247 raft_consensus.cc:686] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.628567 26083 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.628443 25907 catalog_manager.cc:5484] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 1 to 2, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 2 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.629462 26247 consensus_queue.cc:237] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.631465 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.632045 26156 raft_consensus.cc:3032] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.632328 26156 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:23.633040 26156 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:23.636967 26156 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.634917 26083 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.637550 26218 consensus_queue.cc:1035] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.639734 25973 leader_election.cc:302] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.642516 25907 catalog_manager.cc:5484] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 1 to 2, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.643749 26156 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.644245 26246 raft_consensus.cc:2781] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.646536 26246 raft_consensus.cc:686] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.647365 26246 consensus_queue.cc:237] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:23.645213 26247 consensus_queue.cc:1035] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.651788 26251 consensus_queue.cc:1035] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.663033 26251 consensus_queue.cc:1035] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.672185 25905 catalog_manager.cc:5484] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 1 to 2, leader changed from 95176e9e43774b968768d071b45fa063 (127.24.194.3) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:23.675078 26233 consensus_queue.cc:1035] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:23.675151 26248 consensus_queue.cc:1035] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:23.678689 26220 raft_consensus.cc:979] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.679184 26220 raft_consensus.cc:1067] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:23.681232 26007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:23.681844 26007 raft_consensus.cc:482] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:23.682320 26007 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.696342 26203 consensus_queue.cc:1035] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:23.697034 26007 raft_consensus.cc:504] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.699606 26007 leader_election.cc:288] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:23.704530 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:23.705260 26156 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.706367 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f0aa5c067f48b2a1cd8bdd734c914c" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:23.707038 26083 raft_consensus.cc:3032] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:23.707347 26083 raft_consensus.cc:729] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:23.708290 26083 consensus_queue.cc:260] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.710014 26083 raft_consensus.cc:3037] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:23.717502 26083 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.718613 25973 leader_election.cc:302] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:23.718660 26156 raft_consensus.cc:2454] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:23.719292 26203 raft_consensus.cc:2781] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:23.719765 26203 raft_consensus.cc:686] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:23.720549 26203 consensus_queue.cc:237] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:23.728127 25905 catalog_manager.cc:5484] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 1 to 2, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:23.954115 26156 raft_consensus.cc:1261] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:23.955351 26248 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:23.962153 26007 raft_consensus.cc:1261] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:23.963762 26251 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.002997 26083 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.004077 26203 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.012118 26156 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.013587 26203 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.024338 26007 raft_consensus.cc:1261] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.025454 26251 consensus_queue.cc:1035] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.032531 26156 raft_consensus.cc:1261] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.033793 26205 consensus_queue.cc:1035] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.079481 26007 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.080649 26205 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.087615 26156 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.088886 26251 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.106098 26156 raft_consensus.cc:1261] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.107134 26203 consensus_queue.cc:1035] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.114143 26083 raft_consensus.cc:1261] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.115335 26247 consensus_queue.cc:1035] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.177119 26156 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.178520 26247 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.185773 26083 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.186990 26246 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.240633 26156 raft_consensus.cc:1261] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.241854 26083 raft_consensus.cc:1261] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:24.241744 26246 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:24.243002 26247 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:26.322013 25352 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240502 08:05:26.322667 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:26.337888 26007 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:26.338441 26007 raft_consensus.cc:595] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Received request to transfer leadership to TS 95176e9e43774b968768d071b45fa063
I20240502 08:05:26.340473 26007 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:26.340942 26007 raft_consensus.cc:595] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Received request to transfer leadership to TS 95176e9e43774b968768d071b45fa063
I20240502 08:05:26.341827 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 08:05:26.342293 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:26.393944 26246 raft_consensus.cc:979] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:26.394434 26246 raft_consensus.cc:1067] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Signalling peer 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:26.395670 26158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:56728
I20240502 08:05:26.396138 26158 raft_consensus.cc:482] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:26.396422 26158 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.400897 26158 raft_consensus.cc:504] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:26.402429 26158 leader_election.cc:288] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 3 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523)
I20240502 08:05:26.403297 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:26.403443 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:26.403940 26007 raft_consensus.cc:3032] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:26.403936 26083 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.404273 26007 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:26.404883 26007 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:26.405779 26007 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.408422 26083 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 3.
I20240502 08:05:26.409350 26122 leader_election.cc:302] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:26.409991 26284 raft_consensus.cc:2781] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:26.410283 26007 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 3.
I20240502 08:05:26.412040 26284 raft_consensus.cc:686] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:26.412919 26284 consensus_queue.cc:237] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:26.419600 25907 catalog_manager.cc:5484] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 2 to 3, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:26.457994 26289 raft_consensus.cc:979] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:26.458446 26246 raft_consensus.cc:1067] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Signalling peer 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:26.459702 26158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:56728
I20240502 08:05:26.460352 26158 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:26.460638 26158 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.465037 26158 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:26.466601 26158 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 3 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:26.467345 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:26.467574 26083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:26.467839 26007 raft_consensus.cc:3032] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:26.468076 26083 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.468122 26007 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:26.468887 26007 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:26.469820 26007 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:26.472960 26083 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 3.
I20240502 08:05:26.473861 26122 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:26.474124 26007 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 3.
I20240502 08:05:26.474572 26284 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:26.475006 26284 raft_consensus.cc:686] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:26.475739 26284 consensus_queue.cc:237] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:26.482213 25907 catalog_manager.cc:5484] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 2 to 3, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:26.843549 26083 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:26.844801 26284 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:26.854075 26007 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:26.855661 26284 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:05:26.988431 26007 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:26.989552 26284 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:26.997705 26083 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:26.999146 26299 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:27.343000 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:27.348577 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:27.349025 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:28.349735 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:28.354995 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:28.355441 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:29.356324 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:29.362012 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:29.362432 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:30.363212 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:30.368647 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:30.369297 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:31.370429 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:31.376858 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:31.377331 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:32.378188 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:32.383417 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:32.383818 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:33.384663 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:33.390434 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:33.390885 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:34.391594 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:34.397042 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:34.397516 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:35.398373 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:35.403755 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:35.404187 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:36.427436 25352 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240502 08:05:36.431454 26156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "95176e9e43774b968768d071b45fa063"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:56730
I20240502 08:05:36.432148 26156 raft_consensus.cc:595] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:36.440595 26156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "95176e9e43774b968768d071b45fa063"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:56730
I20240502 08:05:36.441072 26156 raft_consensus.cc:595] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:36.447290 26082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:36.447751 26082 raft_consensus.cc:595] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:36.451709 26082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:36.452143 26082 raft_consensus.cc:595] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:36.455986 26082 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
mode: GRACEFUL
new_leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:53560
I20240502 08:05:36.456413 26082 raft_consensus.cc:595] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Received request to transfer leadership to TS f3c2fbdd0db749cf8cf558cac1b77e73
I20240502 08:05:36.463179 26314 raft_consensus.cc:979] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.463676 26311 raft_consensus.cc:1067] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.465044 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58538
I20240502 08:05:36.465518 26008 raft_consensus.cc:482] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:36.465868 26008 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.471890 26008 raft_consensus.cc:504] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:36.473739 26008 leader_election.cc:288] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 4 election: Requested vote from peers 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:36.474895 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:36.474697 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0ff0d20ea4e319df305a9712534d5" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:36.475522 26082 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.475593 26158 raft_consensus.cc:3032] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:36.475926 26158 raft_consensus.cc:729] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:36.476570 26158 consensus_queue.cc:260] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:36.477576 26158 raft_consensus.cc:3037] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.480389 26082 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 4.
I20240502 08:05:36.481469 25973 leader_election.cc:302] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:36.482331 26312 raft_consensus.cc:2781] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:36.482939 26158 raft_consensus.cc:2454] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 4.
I20240502 08:05:36.483256 26312 raft_consensus.cc:686] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:36.484148 26312 consensus_queue.cc:237] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } }
I20240502 08:05:36.490231 25905 catalog_manager.cc:5484] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 3 to 4, leader changed from 95176e9e43774b968768d071b45fa063 (127.24.194.3) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 4 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:36.607692 26306 raft_consensus.cc:979] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.608161 26321 raft_consensus.cc:1067] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.609513 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:36.609949 26008 raft_consensus.cc:482] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:36.610247 26008 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.614645 26008 raft_consensus.cc:504] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.615983 26008 leader_election.cc:288] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 3 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:36.616675 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:36.616894 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:36.617305 26082 raft_consensus.cc:3032] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:36.617363 26158 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.617738 26082 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:36.618531 26082 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.619813 26082 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.623708 26158 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 3.
I20240502 08:05:36.624662 25973 leader_election.cc:302] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:36.625275 26312 raft_consensus.cc:2781] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:36.625746 26312 raft_consensus.cc:686] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:36.625993 26082 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 3.
I20240502 08:05:36.626717 26312 consensus_queue.cc:237] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.632886 25905 catalog_manager.cc:5484] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 2 to 3, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 3 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.655855 26306 raft_consensus.cc:979] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.656236 26321 raft_consensus.cc:1067] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.657454 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:36.657892 26008 raft_consensus.cc:482] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:36.658180 26008 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.662227 26008 raft_consensus.cc:504] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.663504 26008 leader_election.cc:288] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 3 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:36.664557 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:36.664788 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5e0423799c4a568e0ab3ba7bd5c594" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:36.665133 26158 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.665597 26082 raft_consensus.cc:3032] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:36.665928 26082 raft_consensus.cc:729] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:36.666612 26082 consensus_queue.cc:260] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.667573 26082 raft_consensus.cc:3037] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:36.670024 26158 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 3.
I20240502 08:05:36.670974 25973 leader_election.cc:302] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:36.671555 26312 raft_consensus.cc:2781] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:36.671902 26312 raft_consensus.cc:686] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:36.672303 26082 raft_consensus.cc:2454] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 3.
I20240502 08:05:36.672680 26312 consensus_queue.cc:237] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.679359 25905 catalog_manager.cc:5484] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 2 to 3, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 3 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.899624 26314 raft_consensus.cc:979] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.900036 26311 raft_consensus.cc:1067] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.901255 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58538
I20240502 08:05:36.901661 26008 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:36.901964 26008 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.906246 26008 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.907478 26008 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 4 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:36.908290 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:36.908463 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:36.908887 26082 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.908914 26158 raft_consensus.cc:3032] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:36.909454 26158 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:36.910262 26158 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.911422 26158 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:36.915699 26082 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 4.
I20240502 08:05:36.916754 25973 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:36.917474 26312 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:36.917647 26158 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 4.
I20240502 08:05:36.919015 26306 raft_consensus.cc:979] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: : Instructing follower f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.918504 26312 raft_consensus.cc:686] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:36.919466 26321 raft_consensus.cc:1067] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Signalling peer f3c2fbdd0db749cf8cf558cac1b77e73 to start an election
I20240502 08:05:36.919931 26312 consensus_queue.cc:237] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.920897 26008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
from {username='slave'} at 127.0.0.1:58540
I20240502 08:05:36.921413 26008 raft_consensus.cc:482] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:36.921764 26008 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:36.926904 26008 raft_consensus.cc:504] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.927345 25905 catalog_manager.cc:5484] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 3 to 4, leader changed from 95176e9e43774b968768d071b45fa063 (127.24.194.3) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 4 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.929718 26158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:36.930284 26158 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:36.930367 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f1cd3bd434b31ac693cc46c0eaaed" candidate_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:36.931013 26082 raft_consensus.cc:3032] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:36.931337 26082 raft_consensus.cc:729] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:36.931803 26008 leader_election.cc:288] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Requested vote from peers 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:36.932296 26082 consensus_queue.cc:260] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.933547 26082 raft_consensus.cc:3037] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:36.935571 26158 raft_consensus.cc:2454] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:36.936286 26156 raft_consensus.cc:1226] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Rejecting Update request from peer 4e83328baa9a4c79891c971e0cbd6e44 for earlier term 1. Current term is 2. Ops: []
I20240502 08:05:36.937443 25973 leader_election.cc:302] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95176e9e43774b968768d071b45fa063, f3c2fbdd0db749cf8cf558cac1b77e73; no voters:
I20240502 08:05:36.938493 26312 raft_consensus.cc:2781] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:36.939134 26312 raft_consensus.cc:686] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Leader. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:36.940189 26082 raft_consensus.cc:2454] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2fbdd0db749cf8cf558cac1b77e73 in term 2.
I20240502 08:05:36.940253 26312 consensus_queue.cc:237] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:36.946779 25905 catalog_manager.cc:5484] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 reported cstate change: term changed from 1 to 2, leader changed from 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2) to f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1). New cstate: current_term: 2 leader_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.969851 26156 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:36.970948 26312 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:36.980048 26082 raft_consensus.cc:1261] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:36.981750 26312 consensus_queue.cc:1035] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:37.018903 26156 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:37.020056 26312 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:37.028637 26082 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:37.031689 26315 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:05:37.101899 26156 raft_consensus.cc:1261] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:37.103111 26315 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:37.113054 26082 raft_consensus.cc:1261] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:37.114423 26332 consensus_queue.cc:1035] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:05:37.363476 26158 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:37.364519 26332 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:37.372728 26082 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:37.374694 26332 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:37.431996 26082 raft_consensus.cc:1261] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:37.433090 26332 consensus_queue.cc:1035] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:37.441177 26158 raft_consensus.cc:1261] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Refusing update from remote peer f3c2fbdd0db749cf8cf558cac1b77e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:37.442975 26332 consensus_queue.cc:1035] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:39.484476 25352 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240502 08:05:39.485116 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:39.492689 26008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:39.493140 26008 raft_consensus.cc:595] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Received request to transfer leadership to TS 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:39.495030 26008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:39.495425 26008 raft_consensus.cc:595] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Received request to transfer leadership to TS 95176e9e43774b968768d071b45fa063
I20240502 08:05:39.497040 26008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:39.497417 26008 raft_consensus.cc:595] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Received request to transfer leadership to TS 95176e9e43774b968768d071b45fa063
I20240502 08:05:39.499228 26008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a"
dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
mode: GRACEFUL
new_leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:58570
I20240502 08:05:39.499702 26008 raft_consensus.cc:595] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Received request to transfer leadership to TS 4e83328baa9a4c79891c971e0cbd6e44
I20240502 08:05:39.500629 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 08:05:39.501088 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:39.625494 26341 raft_consensus.cc:979] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:39.625912 26332 raft_consensus.cc:1067] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Signalling peer 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:39.627271 26156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf"
dest_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:56728
I20240502 08:05:39.627692 26156 raft_consensus.cc:482] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:39.627960 26156 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:39.632405 26156 raft_consensus.cc:504] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.634200 26156 leader_election.cc:288] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 2 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:39.635134 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:39.635130 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2ac7935c48411995c00d1e411b5caf" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:39.635593 26008 raft_consensus.cc:3032] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:39.635810 26082 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:39.635879 26008 raft_consensus.cc:729] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:39.636587 26008 consensus_queue.cc:260] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.637739 26008 raft_consensus.cc:3037] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:39.640636 26082 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 2.
I20240502 08:05:39.641744 26122 leader_election.cc:302] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:39.642513 26311 raft_consensus.cc:2781] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:39.642869 26008 raft_consensus.cc:2454] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 2.
I20240502 08:05:39.643532 26311 raft_consensus.cc:686] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 2 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:39.644414 26311 consensus_queue.cc:237] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.650789 25905 catalog_manager.cc:5484] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 1 to 2, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:39.725078 26341 raft_consensus.cc:979] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:39.725522 26332 raft_consensus.cc:1067] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Signalling peer 95176e9e43774b968768d071b45fa063 to start an election
I20240502 08:05:39.726713 26156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd"
dest_uuid: "95176e9e43774b968768d071b45fa063"
from {username='slave'} at 127.0.0.1:56728
I20240502 08:05:39.727149 26156 raft_consensus.cc:482] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:39.727408 26156 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:39.731498 26156 raft_consensus.cc:504] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.732805 26156 leader_election.cc:288] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 5 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235)
I20240502 08:05:39.733482 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:39.733717 26082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834e718719042dfa1caebe291dfafcd" candidate_uuid: "95176e9e43774b968768d071b45fa063" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
I20240502 08:05:39.733950 26008 raft_consensus.cc:3032] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:05:39.734331 26008 raft_consensus.cc:729] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:39.734340 26082 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:39.734972 26008 consensus_queue.cc:260] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.735882 26008 raft_consensus.cc:3037] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:05:39.738030 26341 raft_consensus.cc:979] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:39.738499 26332 raft_consensus.cc:1067] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Signalling peer 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:39.739694 26082 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 5.
I20240502 08:05:39.740077 26083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:53548
I20240502 08:05:39.740553 26083 raft_consensus.cc:482] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:39.740548 26122 leader_election.cc:302] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:39.741032 26083 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:39.741137 26008 raft_consensus.cc:2454] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95176e9e43774b968768d071b45fa063 in term 5.
I20240502 08:05:39.741477 26311 raft_consensus.cc:2781] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:05:39.741953 26311 raft_consensus.cc:686] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 5 LEADER]: Becoming Leader. State: Replica: 95176e9e43774b968768d071b45fa063, State: Running, Role: LEADER
I20240502 08:05:39.742621 26311 consensus_queue.cc:237] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.746904 26083 raft_consensus.cc:504] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.749235 26083 leader_election.cc:288] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 3 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:39.749379 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:39.749799 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee0db9a1d91441d85d161036d046a8a" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:39.750236 26008 raft_consensus.cc:3032] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:05:39.750353 26156 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:39.750782 26008 raft_consensus.cc:729] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:39.750211 25905 catalog_manager.cc:5484] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 reported cstate change: term changed from 4 to 5, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 95176e9e43774b968768d071b45fa063 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "95176e9e43774b968768d071b45fa063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:39.751569 26008 consensus_queue.cc:260] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.752712 26008 raft_consensus.cc:3037] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:05:39.756510 26156 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 3.
I20240502 08:05:39.757839 26048 leader_election.cc:302] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:39.758565 26321 raft_consensus.cc:2781] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:05:39.760176 26008 raft_consensus.cc:2454] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 3.
I20240502 08:05:39.762179 26321 raft_consensus.cc:686] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:39.763224 26321 consensus_queue.cc:237] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.770393 25905 catalog_manager.cc:5484] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 2 to 3, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 3 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:39.906695 26341 raft_consensus.cc:979] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: : Instructing follower 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:39.907111 26332 raft_consensus.cc:1067] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Signalling peer 4e83328baa9a4c79891c971e0cbd6e44 to start an election
I20240502 08:05:39.908393 26083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb"
dest_uuid: "4e83328baa9a4c79891c971e0cbd6e44"
from {username='slave'} at 127.0.0.1:53548
I20240502 08:05:39.908821 26083 raft_consensus.cc:482] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:39.909083 26083 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:39.913275 26083 raft_consensus.cc:504] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.914561 26083 leader_election.cc:288] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 4 election: Requested vote from peers f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523), 95176e9e43774b968768d071b45fa063 (127.24.194.3:36727)
I20240502 08:05:39.915313 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73"
I20240502 08:05:39.915532 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0232ea64cb9a46c38908f3405e4625cb" candidate_uuid: "4e83328baa9a4c79891c971e0cbd6e44" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "95176e9e43774b968768d071b45fa063"
I20240502 08:05:39.915786 26008 raft_consensus.cc:3032] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:05:39.915978 26156 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:39.916064 26008 raft_consensus.cc:729] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f3c2fbdd0db749cf8cf558cac1b77e73, State: Running, Role: LEADER
I20240502 08:05:39.916667 26008 consensus_queue.cc:260] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.917457 26008 raft_consensus.cc:3037] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:05:39.920068 26156 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 4.
I20240502 08:05:39.920905 26048 leader_election.cc:302] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e83328baa9a4c79891c971e0cbd6e44, 95176e9e43774b968768d071b45fa063; no voters:
I20240502 08:05:39.921291 26008 raft_consensus.cc:2454] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e83328baa9a4c79891c971e0cbd6e44 in term 4.
I20240502 08:05:39.921574 26321 raft_consensus.cc:2781] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:05:39.922062 26321 raft_consensus.cc:686] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 LEADER]: Becoming Leader. State: Replica: 4e83328baa9a4c79891c971e0cbd6e44, State: Running, Role: LEADER
I20240502 08:05:39.922816 26321 consensus_queue.cc:237] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } }
I20240502 08:05:39.928611 25905 catalog_manager.cc:5484] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 reported cstate change: term changed from 3 to 4, leader changed from f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1) to 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2). New cstate: current_term: 4 leader_uuid: "4e83328baa9a4c79891c971e0cbd6e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:40.120373 26083 raft_consensus.cc:1261] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:40.121562 26311 consensus_queue.cc:1035] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:05:40.131389 26008 raft_consensus.cc:1261] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:05:40.132999 26311 consensus_queue.cc:1035] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:05:40.185906 26008 raft_consensus.cc:1261] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:40.186960 26321 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:40.192257 26008 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 5 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:40.194252 26311 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:40.197399 26156 raft_consensus.cc:1261] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:05:40.199054 26321 consensus_queue.cc:1035] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:05:40.205929 26083 raft_consensus.cc:1261] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 5 FOLLOWER]: Refusing update from remote peer 95176e9e43774b968768d071b45fa063: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 08:05:40.208405 26311 consensus_queue.cc:1035] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e83328baa9a4c79891c971e0cbd6e44" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:05:40.420627 26007 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:40.421689 26321 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2fbdd0db749cf8cf558cac1b77e73" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:40.429970 26156 raft_consensus.cc:1261] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Refusing update from remote peer 4e83328baa9a4c79891c971e0cbd6e44: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:05:40.431416 26321 consensus_queue.cc:1035] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95176e9e43774b968768d071b45fa063" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:40.501878 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:40.507350 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:40.507705 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:41.508548 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:41.516117 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:41.516547 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:42.517335 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:42.522203 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:42.522542 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:43.523320 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:43.528589 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:43.528975 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:44.529688 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:44.534817 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:44.535203 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:45.535989 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:45.541749 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:45.542224 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:46.542987 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:46.548318 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:46.548705 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:47.549510 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:47.554452 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:47.554860 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:48.555657 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:05:48.560815 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:05:48.561208 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:05:49.586786 25352 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240502 08:05:49.587484 25352 tablet_server.cc:178] TabletServer@127.24.194.1:0 shutting down...
I20240502 08:05:49.618230 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:49.618839 25352 tablet_replica.cc:330] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.619534 25352 raft_consensus.cc:2229] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.620795 25352 raft_consensus.cc:2258] T 952f1cd3bd434b31ac693cc46c0eaaed P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.623103 25352 tablet_replica.cc:330] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.623601 25352 raft_consensus.cc:2229] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.624017 25352 raft_consensus.cc:2258] T bf058a946c04431eb4146726bbf044d7 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.625593 25352 tablet_replica.cc:330] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.626093 25352 raft_consensus.cc:2229] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.626503 25352 raft_consensus.cc:2258] T 7e9767b330e7404383f087c7d44bd119 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.628433 25352 tablet_replica.cc:330] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.628903 25352 raft_consensus.cc:2229] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.629308 25352 raft_consensus.cc:2258] T eee0db9a1d91441d85d161036d046a8a P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.631018 25352 tablet_replica.cc:330] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.631496 25352 raft_consensus.cc:2229] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.631888 25352 raft_consensus.cc:2258] T 0232ea64cb9a46c38908f3405e4625cb P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.633394 25352 tablet_replica.cc:330] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.633924 25352 raft_consensus.cc:2229] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.634394 25352 raft_consensus.cc:2258] T 0d2ac7935c48411995c00d1e411b5caf P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.635970 25352 tablet_replica.cc:330] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.636408 25352 raft_consensus.cc:2229] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.637231 25352 raft_consensus.cc:2258] T c9f0aa5c067f48b2a1cd8bdd734c914c P f3c2fbdd0db749cf8cf558cac1b77e73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.638885 25352 tablet_replica.cc:330] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.639354 25352 raft_consensus.cc:2229] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.640153 25352 raft_consensus.cc:2258] T bd5e0423799c4a568e0ab3ba7bd5c594 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.641836 25352 tablet_replica.cc:330] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.642371 25352 raft_consensus.cc:2229] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.643112 25352 raft_consensus.cc:2258] T 16d0ff0d20ea4e319df305a9712534d5 P f3c2fbdd0db749cf8cf558cac1b77e73 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.644762 25352 tablet_replica.cc:330] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73: stopping tablet replica
I20240502 08:05:49.645192 25352 raft_consensus.cc:2229] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.645558 25352 raft_consensus.cc:2258] T 0834e718719042dfa1caebe291dfafcd P f3c2fbdd0db749cf8cf558cac1b77e73 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.676162 25352 tablet_server.cc:194] TabletServer@127.24.194.1:0 shutdown complete.
I20240502 08:05:49.693758 25352 tablet_server.cc:178] TabletServer@127.24.194.2:0 shutting down...
W20240502 08:05:49.719525 26122 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 08:05:49.723343 26122 consensus_peers.cc:485] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 -> Peer 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235): Couldn't send request to peer 4e83328baa9a4c79891c971e0cbd6e44. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:49.723665 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:49.724418 25352 tablet_replica.cc:330] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.725427 25352 raft_consensus.cc:2229] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.726029 25352 raft_consensus.cc:2258] T 16d0ff0d20ea4e319df305a9712534d5 P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.728009 25352 tablet_replica.cc:330] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.728506 25352 raft_consensus.cc:2229] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.729451 25352 raft_consensus.cc:2258] T 7e9767b330e7404383f087c7d44bd119 P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.731578 25352 tablet_replica.cc:330] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.732019 25352 raft_consensus.cc:2229] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.732378 25352 raft_consensus.cc:2258] T 0d2ac7935c48411995c00d1e411b5caf P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.733943 25352 tablet_replica.cc:330] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.734426 25352 raft_consensus.cc:2229] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.735143 25352 raft_consensus.cc:2258] T 0232ea64cb9a46c38908f3405e4625cb P 4e83328baa9a4c79891c971e0cbd6e44 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.736626 25352 tablet_replica.cc:330] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.737044 25352 raft_consensus.cc:2229] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.737381 25352 raft_consensus.cc:2258] T 0834e718719042dfa1caebe291dfafcd P 4e83328baa9a4c79891c971e0cbd6e44 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.738961 25352 tablet_replica.cc:330] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.739391 25352 raft_consensus.cc:2229] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.739715 25352 raft_consensus.cc:2258] T 952f1cd3bd434b31ac693cc46c0eaaed P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.741228 25352 tablet_replica.cc:330] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.741708 25352 raft_consensus.cc:2229] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.742154 25352 raft_consensus.cc:2258] T bf058a946c04431eb4146726bbf044d7 P 4e83328baa9a4c79891c971e0cbd6e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.743760 25352 tablet_replica.cc:330] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.744261 25352 raft_consensus.cc:2229] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.744639 25352 raft_consensus.cc:2258] T bd5e0423799c4a568e0ab3ba7bd5c594 P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.746371 25352 tablet_replica.cc:330] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.746780 25352 raft_consensus.cc:2229] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.747138 25352 raft_consensus.cc:2258] T c9f0aa5c067f48b2a1cd8bdd734c914c P 4e83328baa9a4c79891c971e0cbd6e44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.748584 25352 tablet_replica.cc:330] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44: stopping tablet replica
I20240502 08:05:49.748970 25352 raft_consensus.cc:2229] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.749622 25352 raft_consensus.cc:2258] T eee0db9a1d91441d85d161036d046a8a P 4e83328baa9a4c79891c971e0cbd6e44 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.777693 25352 tablet_server.cc:194] TabletServer@127.24.194.2:0 shutdown complete.
I20240502 08:05:49.794972 25352 tablet_server.cc:178] TabletServer@127.24.194.3:0 shutting down...
W20240502 08:05:49.818194 26124 consensus_peers.cc:485] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 -> Peer f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523): Couldn't send request to peer f3c2fbdd0db749cf8cf558cac1b77e73. Status: Network error: Client connection negotiation failed: client connection to 127.24.194.1:37523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:49.818677 26124 consensus_peers.cc:485] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 -> Peer f3c2fbdd0db749cf8cf558cac1b77e73 (127.24.194.1:37523): Couldn't send request to peer f3c2fbdd0db749cf8cf558cac1b77e73. Status: Network error: Client connection negotiation failed: client connection to 127.24.194.1:37523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:49.825156 26122 consensus_peers.cc:485] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 -> Peer 4e83328baa9a4c79891c971e0cbd6e44 (127.24.194.2:46235): Couldn't send request to peer 4e83328baa9a4c79891c971e0cbd6e44. Status: Network error: Client connection negotiation failed: client connection to 127.24.194.2:46235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:49.825503 25352 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:05:49.826301 25352 tablet_replica.cc:330] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.826911 25352 raft_consensus.cc:2229] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.827350 25352 raft_consensus.cc:2258] T 952f1cd3bd434b31ac693cc46c0eaaed P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.829083 25352 tablet_replica.cc:330] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.829591 25352 raft_consensus.cc:2229] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.830368 25352 raft_consensus.cc:2258] T bf058a946c04431eb4146726bbf044d7 P 95176e9e43774b968768d071b45fa063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.832026 25352 tablet_replica.cc:330] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.832507 25352 raft_consensus.cc:2229] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.832895 25352 raft_consensus.cc:2258] T eee0db9a1d91441d85d161036d046a8a P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.834580 25352 tablet_replica.cc:330] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.835085 25352 raft_consensus.cc:2229] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 5 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.835805 25352 raft_consensus.cc:2258] T 0834e718719042dfa1caebe291dfafcd P 95176e9e43774b968768d071b45fa063 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.837337 25352 tablet_replica.cc:330] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.837800 25352 raft_consensus.cc:2229] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.838227 25352 raft_consensus.cc:2258] T 7e9767b330e7404383f087c7d44bd119 P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.839767 25352 tablet_replica.cc:330] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.840220 25352 raft_consensus.cc:2229] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.840597 25352 raft_consensus.cc:2258] T c9f0aa5c067f48b2a1cd8bdd734c914c P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.842236 25352 tablet_replica.cc:330] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.842695 25352 raft_consensus.cc:2229] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.843091 25352 raft_consensus.cc:2258] T bd5e0423799c4a568e0ab3ba7bd5c594 P 95176e9e43774b968768d071b45fa063 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.844646 25352 tablet_replica.cc:330] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.845077 25352 raft_consensus.cc:2229] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.845517 25352 raft_consensus.cc:2258] T 0232ea64cb9a46c38908f3405e4625cb P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.847038 25352 tablet_replica.cc:330] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.847436 25352 raft_consensus.cc:2229] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.848119 25352 raft_consensus.cc:2258] T 0d2ac7935c48411995c00d1e411b5caf P 95176e9e43774b968768d071b45fa063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.849617 25352 tablet_replica.cc:330] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063: stopping tablet replica
I20240502 08:05:49.850005 25352 raft_consensus.cc:2229] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.850378 25352 raft_consensus.cc:2258] T 16d0ff0d20ea4e319df305a9712534d5 P 95176e9e43774b968768d071b45fa063 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.877874 25352 tablet_server.cc:194] TabletServer@127.24.194.3:0 shutdown complete.
I20240502 08:05:49.895457 25352 master.cc:537] Master@127.24.194.62:44307 shutting down...
I20240502 08:05:49.919183 25352 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:05:49.919677 25352 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.919947 25352 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2883e98f465a4cb88f46da43c40ab55b: stopping tablet replica
I20240502 08:05:49.937104 25352 master.cc:552] Master@127.24.194.62:44307 shutdown complete.
4e83328baa9a4c79891c971e0cbd6e44 4
95176e9e43774b968768d071b45fa063 1
f3c2fbdd0db749cf8cf558cac1b77e73 5
4e83328baa9a4c79891c971e0cbd6e44 4
95176e9e43774b968768d071b45fa063 3
f3c2fbdd0db749cf8cf558cac1b77e73 3
4e83328baa9a4c79891c971e0cbd6e44 1
95176e9e43774b968768d071b45fa063 1
f3c2fbdd0db749cf8cf558cac1b77e73 8
4e83328baa9a4c79891c971e0cbd6e44 3
95176e9e43774b968768d071b45fa063 3
f3c2fbdd0db749cf8cf558cac1b77e73 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28190 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240502 08:05:49.978521 25352 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.194.62:42981
I20240502 08:05:49.979635 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:49.985584 26374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:49.986243 26375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:49.987181 25352 server_base.cc:1034] running on GCE node
W20240502 08:05:49.987913 26377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:49.988705 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:49.988898 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:49.989022 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637149989011 us; error 0 us; skew 500 ppm
I20240502 08:05:49.989490 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:49.992025 25352 webserver.cc:460] Webserver started at http://127.24.194.62:42441/ using document root <none> and password file <none>
I20240502 08:05:49.992460 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:49.992609 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:49.992843 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:49.994066 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/master-0-root/instance:
uuid: "8d0dec82658b4326be63df825d64209e"
format_stamp: "Formatted at 2024-05-02 08:05:49 on dist-test-slave-753s"
I20240502 08:05:49.998683 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:05:50.002195 26382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.003121 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:50.003422 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/master-0-root
uuid: "8d0dec82658b4326be63df825d64209e"
format_stamp: "Formatted at 2024-05-02 08:05:49 on dist-test-slave-753s"
I20240502 08:05:50.003698 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:50.017107 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:50.018342 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:50.055671 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.62:42981
I20240502 08:05:50.055788 26433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.62:42981 every 8 connection(s)
I20240502 08:05:50.059590 26434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.069415 26434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: Bootstrap starting.
I20240502 08:05:50.073676 26434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:50.077925 26434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: No bootstrap required, opened a new log
I20240502 08:05:50.079913 26434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER }
I20240502 08:05:50.080390 26434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:50.080571 26434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0dec82658b4326be63df825d64209e, State: Initialized, Role: FOLLOWER
I20240502 08:05:50.081019 26434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER }
I20240502 08:05:50.081446 26434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:50.081631 26434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:50.081844 26434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:50.085762 26434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER }
I20240502 08:05:50.086269 26434 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0dec82658b4326be63df825d64209e; no voters:
I20240502 08:05:50.087289 26434 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:50.087664 26437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:50.088950 26437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [term 1 LEADER]: Becoming Leader. State: Replica: 8d0dec82658b4326be63df825d64209e, State: Running, Role: LEADER
I20240502 08:05:50.089540 26437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER }
I20240502 08:05:50.090167 26434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:05:50.093690 26438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0dec82658b4326be63df825d64209e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER } }
I20240502 08:05:50.094429 26438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:50.094811 26439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0dec82658b4326be63df825d64209e. Latest consensus state: current_term: 1 leader_uuid: "8d0dec82658b4326be63df825d64209e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0dec82658b4326be63df825d64209e" member_type: VOTER } }
I20240502 08:05:50.095760 26439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:50.097095 26444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:05:50.101720 26444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:05:50.106557 25352 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:05:50.110420 26444 catalog_manager.cc:1347] Generated new cluster ID: a6755316b33644d8b6bc01b0842c7e37
I20240502 08:05:50.110702 26444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:05:50.133643 26444 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:05:50.134882 26444 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:05:50.150645 26444 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: Generated new TSK 0
I20240502 08:05:50.151262 26444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:05:50.173728 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:50.179680 26455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:50.180600 26456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:50.181681 26458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:50.182940 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:50.183751 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:50.183933 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:50.184055 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637150184044 us; error 0 us; skew 500 ppm
I20240502 08:05:50.184562 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:50.186825 25352 webserver.cc:460] Webserver started at http://127.24.194.1:40107/ using document root <none> and password file <none>
I20240502 08:05:50.187252 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:50.187403 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:50.187604 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:50.188586 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root/instance:
uuid: "8f9fe1f3763547388c4301b13a127aaa"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.192885 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:05:50.195924 26463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.196614 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:50.196866 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root
uuid: "8f9fe1f3763547388c4301b13a127aaa"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.197109 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:50.220315 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:50.221475 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:50.222926 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:50.225143 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:50.225314 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.225579 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:50.225785 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.264912 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.1:40577
I20240502 08:05:50.264984 26525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.1:40577 every 8 connection(s)
I20240502 08:05:50.269452 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:50.277583 26530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:50.278605 26531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:50.282392 26526 heartbeater.cc:347] Connected to a master server at 127.24.194.62:42981
W20240502 08:05:50.282529 26533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:50.282797 26526 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:50.283679 25352 server_base.cc:1034] running on GCE node
I20240502 08:05:50.283682 26526 heartbeater.cc:511] Master 127.24.194.62:42981 requested a full tablet report, sending...
I20240502 08:05:50.285020 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:50.285292 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:50.285491 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637150285473 us; error 0 us; skew 500 ppm
I20240502 08:05:50.286016 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:50.286146 26399 ts_manager.cc:194] Registered new tserver with Master: 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:05:50.288043 26399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39264
I20240502 08:05:50.288627 25352 webserver.cc:460] Webserver started at http://127.24.194.2:39777/ using document root <none> and password file <none>
I20240502 08:05:50.289212 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:50.289458 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:50.289775 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:50.291152 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root/instance:
uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.295418 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:05:50.298656 26538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.299499 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:50.299758 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root
uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.300014 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:50.312155 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:50.313390 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:50.314779 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:50.316860 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:50.317049 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.317281 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:50.317431 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.356711 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.2:40601
I20240502 08:05:50.356799 26600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.2:40601 every 8 connection(s)
I20240502 08:05:50.361378 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:50.367949 26604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:50.369258 26605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:50.373046 26601 heartbeater.cc:347] Connected to a master server at 127.24.194.62:42981
I20240502 08:05:50.373488 26601 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:50.374334 26601 heartbeater.cc:511] Master 127.24.194.62:42981 requested a full tablet report, sending...
I20240502 08:05:50.374974 25352 server_base.cc:1034] running on GCE node
W20240502 08:05:50.375638 26607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:50.376610 26399 ts_manager.cc:194] Registered new tserver with Master: 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:50.376636 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:05:50.376958 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:05:50.377135 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637150377117 us; error 0 us; skew 500 ppm
I20240502 08:05:50.377682 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:05:50.377943 26399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39272
I20240502 08:05:50.380642 25352 webserver.cc:460] Webserver started at http://127.24.194.3:33089/ using document root <none> and password file <none>
I20240502 08:05:50.381255 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:50.381451 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:50.381698 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:50.382886 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/instance:
uuid: "755f35bc66a841f382d1c4df6c6f6b64"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.387121 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:05:50.390311 26612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.391101 25352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:50.391387 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root
uuid: "755f35bc66a841f382d1c4df6c6f6b64"
format_stamp: "Formatted at 2024-05-02 08:05:50 on dist-test-slave-753s"
I20240502 08:05:50.391659 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:50.406632 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:50.407735 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:50.409116 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:50.411291 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:50.411481 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.411710 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:50.411870 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:50.450238 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.3:40473
I20240502 08:05:50.450335 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.3:40473 every 8 connection(s)
I20240502 08:05:50.463697 26675 heartbeater.cc:347] Connected to a master server at 127.24.194.62:42981
I20240502 08:05:50.464074 26675 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:50.464826 26675 heartbeater.cc:511] Master 127.24.194.62:42981 requested a full tablet report, sending...
I20240502 08:05:50.466552 26399 ts_manager.cc:194] Registered new tserver with Master: 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:50.466814 25352 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013407643s
I20240502 08:05:50.467829 26399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39274
I20240502 08:05:50.469154 25352 test_util.cc:268] Using random seed: 306560011
I20240502 08:05:50.491595 26399 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39280:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:05:50.497753 26399 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:50.699133 26564 tablet_service.cc:1467] Processing CreateTablet for tablet 6aba3e3cbd94435eabac55bb87284652 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 08:05:50.700726 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aba3e3cbd94435eabac55bb87284652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.701049 26566 tablet_service.cc:1467] Processing CreateTablet for tablet 5145fc92dff54e38b92911a9a14b978e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 08:05:50.704510 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab970a1718349829f0f76fa212d5594 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 08:05:50.706728 26562 tablet_service.cc:1467] Processing CreateTablet for tablet 6808b161f6ae43fda539a49cf1b811c4 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 08:05:50.708914 26561 tablet_service.cc:1467] Processing CreateTablet for tablet e13bc17ba6de4c92a8744b92b7cd3c97 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 08:05:50.699056 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2d8b29e0844df3a3267e030a5aa644 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 08:05:50.713084 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13bc17ba6de4c92a8744b92b7cd3c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.714705 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6808b161f6ae43fda539a49cf1b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.715379 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5145fc92dff54e38b92911a9a14b978e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.715216 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 6aba3e3cbd94435eabac55bb87284652 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 08:05:50.716001 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab970a1718349829f0f76fa212d5594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.715752 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 0b344ca7b5ec4def84739ae6b81fa851 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 08:05:50.716578 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aba3e3cbd94435eabac55bb87284652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.717363 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2d8b29e0844df3a3267e030a5aa644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.717983 26632 tablet_service.cc:1467] Processing CreateTablet for tablet 147d10cad3be4380a85513c7cbaef19e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 08:05:50.719318 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d10cad3be4380a85513c7cbaef19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.719200 26637 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab970a1718349829f0f76fa212d5594 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 08:05:50.720453 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab970a1718349829f0f76fa212d5594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.723194 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 6808b161f6ae43fda539a49cf1b811c4 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 08:05:50.723567 26631 tablet_service.cc:1467] Processing CreateTablet for tablet fcfba0fadba745b3ad79a60a2faccabb (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 08:05:50.724469 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6808b161f6ae43fda539a49cf1b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.725083 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfba0fadba745b3ad79a60a2faccabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.726338 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b344ca7b5ec4def84739ae6b81fa851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.727973 26640 tablet_service.cc:1467] Processing CreateTablet for tablet 5145fc92dff54e38b92911a9a14b978e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 08:05:50.729302 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5145fc92dff54e38b92911a9a14b978e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.731089 26634 tablet_service.cc:1467] Processing CreateTablet for tablet 4efaf1504b1842b3a948e6e10898efc5 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 08:05:50.732334 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efaf1504b1842b3a948e6e10898efc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.734902 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2d8b29e0844df3a3267e030a5aa644 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 08:05:50.736292 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2d8b29e0844df3a3267e030a5aa644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.726514 26635 tablet_service.cc:1467] Processing CreateTablet for tablet e13bc17ba6de4c92a8744b92b7cd3c97 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 08:05:50.712900 26560 tablet_service.cc:1467] Processing CreateTablet for tablet 4efaf1504b1842b3a948e6e10898efc5 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 08:05:50.738942 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efaf1504b1842b3a948e6e10898efc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.740702 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13bc17ba6de4c92a8744b92b7cd3c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.744801 26559 tablet_service.cc:1467] Processing CreateTablet for tablet 0b344ca7b5ec4def84739ae6b81fa851 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 08:05:50.748090 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 147d10cad3be4380a85513c7cbaef19e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 08:05:50.749734 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d10cad3be4380a85513c7cbaef19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.751578 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b344ca7b5ec4def84739ae6b81fa851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.768293 26491 tablet_service.cc:1467] Processing CreateTablet for tablet 5145fc92dff54e38b92911a9a14b978e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 08:05:50.770112 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5145fc92dff54e38b92911a9a14b978e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.779125 26690 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:50.779793 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2d8b29e0844df3a3267e030a5aa644 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 08:05:50.781153 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2d8b29e0844df3a3267e030a5aa644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.784965 26690 tablet_bootstrap.cc:654] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:50.791671 26557 tablet_service.cc:1467] Processing CreateTablet for tablet 77b2256f7eb5420792d8042fb8e30d0b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 08:05:50.791697 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 6aba3e3cbd94435eabac55bb87284652 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 08:05:50.793068 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b2256f7eb5420792d8042fb8e30d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.793192 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aba3e3cbd94435eabac55bb87284652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.807042 26564 tablet_service.cc:1467] Processing CreateTablet for tablet fcfba0fadba745b3ad79a60a2faccabb (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 08:05:50.808365 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfba0fadba745b3ad79a60a2faccabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.817342 26635 tablet_service.cc:1467] Processing CreateTablet for tablet 77b2256f7eb5420792d8042fb8e30d0b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 08:05:50.820418 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 0774e7aa819c48f9bcda1d69e4e88a79 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 08:05:50.827322 26637 tablet_service.cc:1467] Processing CreateTablet for tablet b5fa1083c65a4f999b1b52753bce70ed (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 08:05:50.830148 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 0774e7aa819c48f9bcda1d69e4e88a79 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 08:05:50.829877 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b2256f7eb5420792d8042fb8e30d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.831657 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0774e7aa819c48f9bcda1d69e4e88a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.833385 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0774e7aa819c48f9bcda1d69e4e88a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.834017 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fa1083c65a4f999b1b52753bce70ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.855780 26632 tablet_service.cc:1467] Processing CreateTablet for tablet 33e1db2e4cc44834966a5a186294dff8 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 08:05:50.857105 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e1db2e4cc44834966a5a186294dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.863655 26488 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab970a1718349829f0f76fa212d5594 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 08:05:50.865022 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab970a1718349829f0f76fa212d5594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.866137 26482 tablet_service.cc:1467] Processing CreateTablet for tablet fcfba0fadba745b3ad79a60a2faccabb (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 08:05:50.867542 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfba0fadba745b3ad79a60a2faccabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.867228 26487 tablet_service.cc:1467] Processing CreateTablet for tablet 6808b161f6ae43fda539a49cf1b811c4 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 08:05:50.868584 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6808b161f6ae43fda539a49cf1b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.876272 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 0b344ca7b5ec4def84739ae6b81fa851 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 08:05:50.877600 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b344ca7b5ec4def84739ae6b81fa851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.879036 26483 tablet_service.cc:1467] Processing CreateTablet for tablet 147d10cad3be4380a85513c7cbaef19e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 08:05:50.879742 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 6f68e2a58df64f808deb0f0d3e63e43d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 08:05:50.880357 26483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d10cad3be4380a85513c7cbaef19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.880996 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f68e2a58df64f808deb0f0d3e63e43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.871964 26486 tablet_service.cc:1467] Processing CreateTablet for tablet e13bc17ba6de4c92a8744b92b7cd3c97 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 08:05:50.884532 26566 tablet_service.cc:1467] Processing CreateTablet for tablet bcad69f7daa74587b431486e4cad620f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 08:05:50.876274 26485 tablet_service.cc:1467] Processing CreateTablet for tablet 4efaf1504b1842b3a948e6e10898efc5 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 08:05:50.886521 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 39e50c6374394235996663b59b4be48a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 08:05:50.887192 26635 tablet_service.cc:1467] Processing CreateTablet for tablet bcad69f7daa74587b431486e4cad620f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 08:05:50.888211 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13bc17ba6de4c92a8744b92b7cd3c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.887782 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e50c6374394235996663b59b4be48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.886955 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcad69f7daa74587b431486e4cad620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.887878 26560 tablet_service.cc:1467] Processing CreateTablet for tablet 39e50c6374394235996663b59b4be48a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 08:05:50.889034 26562 tablet_service.cc:1467] Processing CreateTablet for tablet b5fa1083c65a4f999b1b52753bce70ed (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 08:05:50.893491 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efaf1504b1842b3a948e6e10898efc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.864748 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 6f68e2a58df64f808deb0f0d3e63e43d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 08:05:50.906258 26690 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:50.906752 26690 ts_tablet_manager.cc:1367] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.021s
I20240502 08:05:50.908294 26691 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:50.909489 26690 raft_consensus.cc:348] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:50.910240 26690 raft_consensus.cc:374] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:50.910526 26690 raft_consensus.cc:729] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:50.914764 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e50c6374394235996663b59b4be48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.922891 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f68e2a58df64f808deb0f0d3e63e43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.926692 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 33e1db2e4cc44834966a5a186294dff8 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 08:05:50.927975 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e1db2e4cc44834966a5a186294dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.932446 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 77b2256f7eb5420792d8042fb8e30d0b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 08:05:50.936021 26485 tablet_service.cc:1467] Processing CreateTablet for tablet 0774e7aa819c48f9bcda1d69e4e88a79 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 08:05:50.940558 26491 tablet_service.cc:1467] Processing CreateTablet for tablet b5fa1083c65a4f999b1b52753bce70ed (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 08:05:50.944454 26631 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f57a1a34420e98d2d4d7f2f9162e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 08:05:50.946643 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3c527a35284f68b946ea4bef6182c9 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 08:05:50.947048 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 33e1db2e4cc44834966a5a186294dff8 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 08:05:50.956305 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 9a26db3828484577a5da525732f0b102 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 08:05:50.957110 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b2256f7eb5420792d8042fb8e30d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.957501 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 39e50c6374394235996663b59b4be48a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 08:05:50.982594 26566 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3c527a35284f68b946ea4bef6182c9 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 08:05:50.994225 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e1db2e4cc44834966a5a186294dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.995887 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e50c6374394235996663b59b4be48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.991017 26690 consensus_queue.cc:260] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.039309 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0774e7aa819c48f9bcda1d69e4e88a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.990872 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcad69f7daa74587b431486e4cad620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:50.992208 26691 tablet_bootstrap.cc:654] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.007179 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fa1083c65a4f999b1b52753bce70ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.059710 26693 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:51.061235 26487 tablet_service.cc:1467] Processing CreateTablet for tablet bcad69f7daa74587b431486e4cad620f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 08:05:51.063534 26482 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3c527a35284f68b946ea4bef6182c9 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 08:05:51.066231 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 90ed3fca6dd84c87bc322d121bdeff7b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 08:05:51.069386 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fa1083c65a4f999b1b52753bce70ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.070130 26483 tablet_service.cc:1467] Processing CreateTablet for tablet 6f68e2a58df64f808deb0f0d3e63e43d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 08:05:51.005111 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 9a26db3828484577a5da525732f0b102 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 08:05:51.085522 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f57a1a34420e98d2d4d7f2f9162e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.086599 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a26db3828484577a5da525732f0b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.087224 26486 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f57a1a34420e98d2d4d7f2f9162e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 08:05:51.088341 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3c527a35284f68b946ea4bef6182c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.090286 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a26db3828484577a5da525732f0b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.084416 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcad69f7daa74587b431486e4cad620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.090844 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 90ed3fca6dd84c87bc322d121bdeff7b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 08:05:51.092098 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3c527a35284f68b946ea4bef6182c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.098866 26483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f68e2a58df64f808deb0f0d3e63e43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.092128 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ed3fca6dd84c87bc322d121bdeff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.099627 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f57a1a34420e98d2d4d7f2f9162e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.096114 26693 tablet_bootstrap.cc:654] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.093297 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ed3fca6dd84c87bc322d121bdeff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.095182 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 9a26db3828484577a5da525732f0b102 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
W20240502 08:05:51.101998 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:51.102739 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a26db3828484577a5da525732f0b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.099768 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3c527a35284f68b946ea4bef6182c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.020217 26557 tablet_service.cc:1467] Processing CreateTablet for tablet 12b2f57a1a34420e98d2d4d7f2f9162e (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 08:05:51.111040 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b2f57a1a34420e98d2d4d7f2f9162e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.123288 26691 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:51.123836 26691 ts_tablet_manager.cc:1367] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.216s user 0.027s sys 0.009s
I20240502 08:05:51.126572 26691 raft_consensus.cc:348] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.127305 26691 raft_consensus.cc:374] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.127593 26691 raft_consensus.cc:729] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.150247 26691 consensus_queue.cc:260] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.158102 26601 heartbeater.cc:503] Master 127.24.194.62:42981 was elected leader, sending a full tablet report...
I20240502 08:05:51.158090 26697 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.160627 26488 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 08:05:51.161906 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.158821 26690 ts_tablet_manager.cc:1398] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.252s user 0.014s sys 0.014s
I20240502 08:05:51.168834 26564 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 08:05:51.133738 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 90ed3fca6dd84c87bc322d121bdeff7b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 08:05:51.171568 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 08:05:51.158879 26697 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.183231 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.186893 26639 tablet_service.cc:1467] Processing CreateTablet for tablet a80a7fc6eb5046b78755fc53fcf8f1db (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 08:05:51.188647 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80a7fc6eb5046b78755fc53fcf8f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.190559 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ed3fca6dd84c87bc322d121bdeff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.194608 26566 tablet_service.cc:1467] Processing CreateTablet for tablet a80a7fc6eb5046b78755fc53fcf8f1db (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 08:05:51.202589 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80a7fc6eb5046b78755fc53fcf8f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.208060 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2ed1efbf61440fbb8fbaf3ac77e227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.230294 26697 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:51.238389 26690 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:51.243136 26693 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:51.243652 26693 ts_tablet_manager.cc:1367] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.184s user 0.011s sys 0.008s
I20240502 08:05:51.246325 26693 raft_consensus.cc:348] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.247117 26693 raft_consensus.cc:374] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.247397 26693 raft_consensus.cc:729] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.248054 26693 consensus_queue.cc:260] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.263038 26675 heartbeater.cc:503] Master 127.24.194.62:42981 was elected leader, sending a full tablet report...
I20240502 08:05:51.272181 26691 ts_tablet_manager.cc:1398] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.148s user 0.027s sys 0.054s
I20240502 08:05:51.336580 26693 ts_tablet_manager.cc:1398] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.093s user 0.015s sys 0.008s
I20240502 08:05:51.337738 26691 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:51.318811 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc16fa0dfe47728a16df1a9fd99c84 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 08:05:51.341179 26690 tablet_bootstrap.cc:654] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.341681 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cc16fa0dfe47728a16df1a9fd99c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.338173 26693 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:51.338088 26566 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc16fa0dfe47728a16df1a9fd99c84 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 08:05:51.346166 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cc16fa0dfe47728a16df1a9fd99c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.347332 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbfc58321f544229109ce9344e7000d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 08:05:51.333942 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:51.296078 26485 tablet_service.cc:1467] Processing CreateTablet for tablet a80a7fc6eb5046b78755fc53fcf8f1db (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 08:05:51.350479 26691 tablet_bootstrap.cc:654] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:51.350637 26541 leader_election.cc:341] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.350991 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80a7fc6eb5046b78755fc53fcf8f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.308180 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbfc58321f544229109ce9344e7000d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 08:05:51.302809 26526 heartbeater.cc:503] Master 127.24.194.62:42981 was elected leader, sending a full tablet report...
I20240502 08:05:51.332796 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 82da19e5ed7d415588714d971127c197 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 08:05:51.354323 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da19e5ed7d415588714d971127c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.358458 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbfc58321f544229109ce9344e7000d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:51.361374 26602 tablet.cc:2369] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:05:51.299472 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 82da19e5ed7d415588714d971127c197 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 08:05:51.362679 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da19e5ed7d415588714d971127c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.348909 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbfc58321f544229109ce9344e7000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.335433 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbfc58321f544229109ce9344e7000d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 08:05:51.367182 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbfc58321f544229109ce9344e7000d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.335424 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 82da19e5ed7d415588714d971127c197 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 08:05:51.369765 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82da19e5ed7d415588714d971127c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.380707 26693 tablet_bootstrap.cc:654] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.412550 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 4429915312704856895d00171a6e6151 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 08:05:51.413902 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4429915312704856895d00171a6e6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.416766 26635 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc16fa0dfe47728a16df1a9fd99c84 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 08:05:51.418097 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cc16fa0dfe47728a16df1a9fd99c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.437199 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:51.442477 26542 leader_election.cc:341] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.442960 26542 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:51.443169 26690 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:51.444432 26697 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:51.444896 26690 ts_tablet_manager.cc:1367] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.207s user 0.024s sys 0.021s
I20240502 08:05:51.449246 26703 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.449757 26703 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.451530 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 4429915312704856895d00171a6e6151 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 08:05:51.452896 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4429915312704856895d00171a6e6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.452952 26703 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:51.454748 26631 tablet_service.cc:1467] Processing CreateTablet for tablet 14320b2cc936478c8fe96bc8361ad980 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 08:05:51.456015 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14320b2cc936478c8fe96bc8361ad980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.459787 26485 tablet_service.cc:1467] Processing CreateTablet for tablet 14320b2cc936478c8fe96bc8361ad980 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 08:05:51.462700 26690 raft_consensus.cc:348] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.463649 26690 raft_consensus.cc:374] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.463990 26690 raft_consensus.cc:729] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.466727 26691 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:51.467197 26691 ts_tablet_manager.cc:1367] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.130s user 0.057s sys 0.013s
I20240502 08:05:51.467483 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14320b2cc936478c8fe96bc8361ad980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.469108 26566 tablet_service.cc:1467] Processing CreateTablet for tablet 14320b2cc936478c8fe96bc8361ad980 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 08:05:51.470001 26691 raft_consensus.cc:348] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.470785 26691 raft_consensus.cc:374] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.471079 26691 raft_consensus.cc:729] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.471108 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 4429915312704856895d00171a6e6151 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 08:05:51.473397 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4429915312704856895d00171a6e6151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.471808 26691 consensus_queue.cc:260] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.480684 26691 ts_tablet_manager.cc:1398] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20240502 08:05:51.481675 26691 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:51.482440 26699 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.482959 26699 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.485610 26635 tablet_service.cc:1467] Processing CreateTablet for tablet a345a6e692404520a7f76e5e3a0af8b0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 08:05:51.486544 26699 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:51.495436 26693 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:51.486949 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a345a6e692404520a7f76e5e3a0af8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.498831 26564 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb83d16acf49edbe7b8b774d7d7bca (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 08:05:51.472182 26690 consensus_queue.cc:260] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.507937 26690 ts_tablet_manager.cc:1398] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.063s user 0.008s sys 0.000s
I20240502 08:05:51.509007 26690 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:51.511195 26691 tablet_bootstrap.cc:654] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.501763 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14320b2cc936478c8fe96bc8361ad980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.495803 26561 tablet_service.cc:1467] Processing CreateTablet for tablet a345a6e692404520a7f76e5e3a0af8b0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 08:05:51.503306 26693 ts_tablet_manager.cc:1367] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.165s user 0.021s sys 0.007s
I20240502 08:05:51.519673 26557 tablet_service.cc:1467] Processing CreateTablet for tablet 021cd2b53b784089b3fc08aad322e294 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 08:05:51.521458 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a345a6e692404520a7f76e5e3a0af8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.522203 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021cd2b53b784089b3fc08aad322e294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.522917 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fb83d16acf49edbe7b8b774d7d7bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.527529 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 021cd2b53b784089b3fc08aad322e294 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 08:05:51.529045 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021cd2b53b784089b3fc08aad322e294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.538570 26699 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.538897 26563 tablet_service.cc:1467] Processing CreateTablet for tablet ca82734882074cecbdd750a509db67a0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 08:05:51.539088 26699 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.540163 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca82734882074cecbdd750a509db67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.541077 26699 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:51.544471 26631 tablet_service.cc:1467] Processing CreateTablet for tablet ca82734882074cecbdd750a509db67a0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 08:05:51.545737 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca82734882074cecbdd750a509db67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.550278 26633 tablet_service.cc:1467] Processing CreateTablet for tablet ec54ed4ed93048d4a896c43796e5727d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 08:05:51.551640 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54ed4ed93048d4a896c43796e5727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.495803 26639 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb83d16acf49edbe7b8b774d7d7bca (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 08:05:51.553332 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fb83d16acf49edbe7b8b774d7d7bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.557265 26564 tablet_service.cc:1467] Processing CreateTablet for tablet ec54ed4ed93048d4a896c43796e5727d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 08:05:51.558699 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54ed4ed93048d4a896c43796e5727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.563467 26690 tablet_bootstrap.cc:654] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.566993 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:51.567958 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:51.571121 26562 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00601c8043478f8523c59e70966d50 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 08:05:51.571274 26693 raft_consensus.cc:348] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.571736 26500 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:51.571977 26693 raft_consensus.cc:374] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.572338 26693 raft_consensus.cc:729] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.572469 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00601c8043478f8523c59e70966d50. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:51.573464 26615 leader_election.cc:341] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.574208 26615 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:51.575004 26699 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:51.575348 26699 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:51.575670 26699 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:51.578369 26489 tablet_service.cc:1467] Processing CreateTablet for tablet a345a6e692404520a7f76e5e3a0af8b0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 08:05:51.579831 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a345a6e692404520a7f76e5e3a0af8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.592600 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00601c8043478f8523c59e70966d50 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 08:05:51.593956 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00601c8043478f8523c59e70966d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.604843 26632 tablet_service.cc:1467] Processing CreateTablet for tablet 66aad5aa3c99405abb36667908363887 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 08:05:51.606266 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aad5aa3c99405abb36667908363887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.606666 26566 tablet_service.cc:1467] Processing CreateTablet for tablet 66aad5aa3c99405abb36667908363887 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 08:05:51.606248 26691 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:51.610635 26691 ts_tablet_manager.cc:1367] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.006s
I20240502 08:05:51.612838 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:51.613570 26576 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:51.619421 26616 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:51.620820 26691 raft_consensus.cc:348] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.621625 26691 raft_consensus.cc:374] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.621946 26691 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.622457 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:51.622809 26691 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.623461 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aad5aa3c99405abb36667908363887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.628325 26693 consensus_queue.cc:260] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.629451 26559 tablet_service.cc:1467] Processing CreateTablet for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 08:05:51.647373 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.632499 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 8d53d3e99887443b8967129ddd219819 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 08:05:51.650796 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d53d3e99887443b8967129ddd219819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.634198 26558 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9fcbac5f24ae8b0c8f071ae34229f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 08:05:51.654817 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9fcbac5f24ae8b0c8f071ae34229f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.636097 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 8d53d3e99887443b8967129ddd219819 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 08:05:51.658756 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d53d3e99887443b8967129ddd219819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.637368 26485 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb83d16acf49edbe7b8b774d7d7bca (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 08:05:51.638581 26482 tablet_service.cc:1467] Processing CreateTablet for tablet ec54ed4ed93048d4a896c43796e5727d (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 08:05:51.639011 26484 tablet_service.cc:1467] Processing CreateTablet for tablet ca82734882074cecbdd750a509db67a0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 08:05:51.637809 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 021cd2b53b784089b3fc08aad322e294 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 08:05:51.640462 26483 tablet_service.cc:1467] Processing CreateTablet for tablet 6a00601c8043478f8523c59e70966d50 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 08:05:51.642407 26488 tablet_service.cc:1467] Processing CreateTablet for tablet 66aad5aa3c99405abb36667908363887 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 08:05:51.663542 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fb83d16acf49edbe7b8b774d7d7bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.667309 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca82734882074cecbdd750a509db67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.671345 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aad5aa3c99405abb36667908363887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.675503 26483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00601c8043478f8523c59e70966d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.679129 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54ed4ed93048d4a896c43796e5727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.680292 26699 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.683204 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021cd2b53b784089b3fc08aad322e294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.683171 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:51.683880 26500 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:51.688117 26487 tablet_service.cc:1467] Processing CreateTablet for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 08:05:51.688428 26486 tablet_service.cc:1467] Processing CreateTablet for tablet 8d53d3e99887443b8967129ddd219819 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 08:05:51.689657 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.689554 26699 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:51.690955 26699 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:51.692059 26699 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:51.692411 26699 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:51.706112 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:51.706982 26650 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:51.707437 26489 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9fcbac5f24ae8b0c8f071ae34229f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 08:05:51.708429 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:51.708765 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9fcbac5f24ae8b0c8f071ae34229f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:51.713922 26467 leader_election.cc:341] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.714686 26467 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:51.712982 26699 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.717715 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:05:51.719005 26615 leader_election.cc:341] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.719868 26699 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:51.723338 26693 ts_tablet_manager.cc:1398] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.205s user 0.004s sys 0.004s
I20240502 08:05:51.724339 26693 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:51.719005 26636 tablet_service.cc:1467] Processing CreateTablet for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 08:05:51.727397 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b6ee1a6eb54f00b15b4b47a4069d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.732744 26693 tablet_bootstrap.cc:654] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:51.756031 26386 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 112291us
I20240502 08:05:51.754808 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d53d3e99887443b8967129ddd219819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.750442 26500 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:51.763083 26615 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:51.763128 26638 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9fcbac5f24ae8b0c8f071ae34229f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 08:05:51.764321 26714 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:51.764608 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9fcbac5f24ae8b0c8f071ae34229f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.764796 26714 raft_consensus.cc:686] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
W20240502 08:05:51.754825 26385 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 53458.7us
I20240502 08:05:51.766153 26703 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:51.765802 26714 consensus_queue.cc:237] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.775743 26485 tablet_service.cc:1467] Processing CreateTablet for tablet c477dd390aca4aeabf4f028e82984ae2 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 08:05:51.777559 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477dd390aca4aeabf4f028e82984ae2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:51.778520 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 159289us
W20240502 08:05:51.780067 26616 leader_election.cc:341] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.783175 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:51.784351 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:51.784950 26576 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:51.792193 26616 leader_election.cc:341] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.800561 26576 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:51.806882 26691 ts_tablet_manager.cc:1398] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.196s user 0.007s sys 0.010s
I20240502 08:05:51.808087 26691 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:51.810781 26699 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.811234 26699 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.813848 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:51.815250 26615 leader_election.cc:341] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.815498 26690 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:51.815992 26690 ts_tablet_manager.cc:1367] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.307s user 0.018s sys 0.030s
I20240502 08:05:51.816138 26699 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:51.816569 26693 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:51.817013 26693 ts_tablet_manager.cc:1367] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.009s
I20240502 08:05:51.820199 26691 tablet_bootstrap.cc:654] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.823238 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:51.824610 26616 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:51.825434 26714 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 08:05:51.825768 26616 leader_election.cc:341] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.825907 26714 raft_consensus.cc:686] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:51.826277 26616 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:51.827109 26699 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:51.827070 26714 consensus_queue.cc:237] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.832701 26693 raft_consensus.cc:348] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.833541 26693 raft_consensus.cc:374] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.833842 26693 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.834532 26693 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.836989 26693 ts_tablet_manager.cc:1398] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 08:05:51.837934 26693 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:51.841569 26559 tablet_service.cc:1467] Processing CreateTablet for tablet c477dd390aca4aeabf4f028e82984ae2 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 08:05:51.842937 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477dd390aca4aeabf4f028e82984ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.843035 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 1d753d3fcb9249c28f415af2ab615999 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 08:05:51.843350 26639 tablet_service.cc:1467] Processing CreateTablet for tablet e7ac53bd5d944d0bbc8062cd0ba0550a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 08:05:51.844250 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d753d3fcb9249c28f415af2ab615999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.845587 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ac53bd5d944d0bbc8062cd0ba0550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.847019 26558 tablet_service.cc:1467] Processing CreateTablet for tablet e7ac53bd5d944d0bbc8062cd0ba0550a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 08:05:51.848265 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ac53bd5d944d0bbc8062cd0ba0550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.847932 26489 tablet_service.cc:1467] Processing CreateTablet for tablet f57fbc2401954920998f3ba61e449afd (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 08:05:51.849287 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fbc2401954920998f3ba61e449afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.850821 26690 raft_consensus.cc:348] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.851575 26690 raft_consensus.cc:374] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.851830 26564 tablet_service.cc:1467] Processing CreateTablet for tablet f57fbc2401954920998f3ba61e449afd (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 08:05:51.847019 26482 tablet_service.cc:1467] Processing CreateTablet for tablet e7ac53bd5d944d0bbc8062cd0ba0550a (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 08:05:51.853017 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fbc2401954920998f3ba61e449afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.853273 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ac53bd5d944d0bbc8062cd0ba0550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.847955 26488 tablet_service.cc:1467] Processing CreateTablet for tablet 1d753d3fcb9249c28f415af2ab615999 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 08:05:51.855110 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d753d3fcb9249c28f415af2ab615999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.855010 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 1d753d3fcb9249c28f415af2ab615999 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 08:05:51.856256 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d753d3fcb9249c28f415af2ab615999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.860170 26486 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6f8df55d8c4456ac8d84fdcb919e55 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 08:05:51.860777 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6f8df55d8c4456ac8d84fdcb919e55 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 08:05:51.861369 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6f8df55d8c4456ac8d84fdcb919e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.861979 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6f8df55d8c4456ac8d84fdcb919e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.862174 26632 tablet_service.cc:1467] Processing CreateTablet for tablet f57fbc2401954920998f3ba61e449afd (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 08:05:51.863417 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fbc2401954920998f3ba61e449afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.866375 26693 tablet_bootstrap.cc:654] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.851857 26690 raft_consensus.cc:729] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.893637 26693 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:51.893424 26690 consensus_queue.cc:260] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.894150 26693 ts_tablet_manager.cc:1367] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.000s
I20240502 08:05:51.894076 26631 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6f8df55d8c4456ac8d84fdcb919e55 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 08:05:51.895512 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6f8df55d8c4456ac8d84fdcb919e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.896859 26693 raft_consensus.cc:348] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.898803 26693 raft_consensus.cc:374] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.899163 26693 raft_consensus.cc:729] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.907436 26690 ts_tablet_manager.cc:1398] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.091s user 0.008s sys 0.000s
I20240502 08:05:51.901466 26693 consensus_queue.cc:260] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.908517 26690 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:51.911125 26638 tablet_service.cc:1467] Processing CreateTablet for tablet c477dd390aca4aeabf4f028e82984ae2 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 08:05:51.912386 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477dd390aca4aeabf4f028e82984ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.913795 26693 ts_tablet_manager.cc:1398] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 08:05:51.915331 26693 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:51.916142 26691 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:51.916587 26691 ts_tablet_manager.cc:1367] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.000s
I20240502 08:05:51.920889 26693 tablet_bootstrap.cc:654] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.923724 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 3749d35fdedd4d979e761e8b7f57ad39 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 08:05:51.924378 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 3749d35fdedd4d979e761e8b7f57ad39 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 08:05:51.924997 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3749d35fdedd4d979e761e8b7f57ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.925566 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3749d35fdedd4d979e761e8b7f57ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.928400 26559 tablet_service.cc:1467] Processing CreateTablet for tablet fc069ce0343a4e9b9056d3604d2928df (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 08:05:51.929632 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc069ce0343a4e9b9056d3604d2928df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.932672 26486 tablet_service.cc:1467] Processing CreateTablet for tablet fc069ce0343a4e9b9056d3604d2928df (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 08:05:51.935941 26635 tablet_service.cc:1467] Processing CreateTablet for tablet 3749d35fdedd4d979e761e8b7f57ad39 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 08:05:51.937191 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3749d35fdedd4d979e761e8b7f57ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.938534 26690 tablet_bootstrap.cc:654] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.940656 26488 tablet_service.cc:1467] Processing CreateTablet for tablet a3c12a25498a426290c1ea595c9ac021 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 08:05:51.941205 26565 tablet_service.cc:1467] Processing CreateTablet for tablet a3c12a25498a426290c1ea595c9ac021 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 08:05:51.942592 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c12a25498a426290c1ea595c9ac021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.940946 26397 catalog_manager.cc:5484] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:51.943969 26397 catalog_manager.cc:5484] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:51.946188 26703 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:51.946653 26703 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.948706 26703 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:51.949158 26482 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba760137cc473c9fefe88a2cc261f0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 08:05:51.950558 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c12a25498a426290c1ea595c9ac021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.951498 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:51.952977 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:51.954502 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba760137cc473c9fefe88a2cc261f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:51.954596 26467 leader_election.cc:341] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.954831 26691 raft_consensus.cc:348] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:51.955312 26467 leader_election.cc:341] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:51.955780 26467 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:51.956926 26703 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:51.958077 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc069ce0343a4e9b9056d3604d2928df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.958539 26693 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:51.970712 26693 ts_tablet_manager.cc:1367] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.003s
I20240502 08:05:51.973518 26693 raft_consensus.cc:348] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.974524 26693 raft_consensus.cc:374] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.974821 26690 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:51.974884 26693 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.975466 26690 ts_tablet_manager.cc:1367] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.006s
I20240502 08:05:51.975992 26693 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.978152 26691 raft_consensus.cc:374] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.978474 26691 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.978349 26690 raft_consensus.cc:348] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.979239 26690 raft_consensus.cc:374] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:51.979597 26690 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:51.980321 26690 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.981890 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba760137cc473c9fefe88a2cc261f0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 08:05:51.979264 26691 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:51.984515 26691 ts_tablet_manager.cc:1398] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.068s user 0.005s sys 0.002s
I20240502 08:05:51.985579 26691 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:51.988071 26693 ts_tablet_manager.cc:1398] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.017s user 0.003s sys 0.004s
I20240502 08:05:51.994498 26632 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba760137cc473c9fefe88a2cc261f0 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 08:05:51.995872 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba760137cc473c9fefe88a2cc261f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.998178 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba760137cc473c9fefe88a2cc261f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.002488 26485 tablet_service.cc:1467] Processing CreateTablet for tablet a1a8eccc1a164ecbb7d5ada16f48b6de (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 08:05:52.006256 26487 tablet_service.cc:1467] Processing CreateTablet for tablet d73e56c17a6f4770a16b443ab75ea198 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 08:05:52.007330 26639 tablet_service.cc:1467] Processing CreateTablet for tablet a1a8eccc1a164ecbb7d5ada16f48b6de (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 08:05:52.008628 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a8eccc1a164ecbb7d5ada16f48b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.010998 26563 tablet_service.cc:1467] Processing CreateTablet for tablet a1a8eccc1a164ecbb7d5ada16f48b6de (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 08:05:52.012244 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a8eccc1a164ecbb7d5ada16f48b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.014894 26690 ts_tablet_manager.cc:1398] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20240502 08:05:52.013198 26693 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.015866 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a8eccc1a164ecbb7d5ada16f48b6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.015991 26690 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.018312 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73e56c17a6f4770a16b443ab75ea198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.023880 26691 tablet_bootstrap.cc:654] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:51.959306 26636 tablet_service.cc:1467] Processing CreateTablet for tablet fc069ce0343a4e9b9056d3604d2928df (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 08:05:51.959924 26631 tablet_service.cc:1467] Processing CreateTablet for tablet a3c12a25498a426290c1ea595c9ac021 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 08:05:52.026949 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c12a25498a426290c1ea595c9ac021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.038765 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc069ce0343a4e9b9056d3604d2928df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.044044 26690 tablet_bootstrap.cc:654] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.049798 26632 tablet_service.cc:1467] Processing CreateTablet for tablet d73e56c17a6f4770a16b443ab75ea198 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 08:05:52.051252 26632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73e56c17a6f4770a16b443ab75ea198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.051156 26564 tablet_service.cc:1467] Processing CreateTablet for tablet d73e56c17a6f4770a16b443ab75ea198 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 08:05:52.052561 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73e56c17a6f4770a16b443ab75ea198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.056262 26691 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.056780 26691 ts_tablet_manager.cc:1367] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.000s
I20240502 08:05:52.056849 26697 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.057322 26697 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.059509 26697 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.059625 26691 raft_consensus.cc:348] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.060474 26691 raft_consensus.cc:374] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.060786 26691 raft_consensus.cc:729] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.061583 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.061681 26691 consensus_queue.cc:260] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.064136 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:52.064410 26703 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.064913 26500 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:52.064849 26703 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.066141 26541 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:52.066851 26703 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.067277 26697 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.069026 26693 tablet_bootstrap.cc:654] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.068939 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:52.070686 26467 leader_election.cc:341] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.068935 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.071636 26650 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:52.072742 26467 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.073848 26703 raft_consensus.cc:2781] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.074223 26703 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.074566 26703 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.076992 26488 tablet_service.cc:1467] Processing CreateTablet for tablet 40c1d45639ea4980b7b376d3cd3cc1a3 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 08:05:52.077961 26697 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:52.078308 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c1d45639ea4980b7b376d3cd3cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.080602 26482 tablet_service.cc:1467] Processing CreateTablet for tablet 99e8fc1030084a55a81726683707e539 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 08:05:52.081899 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e8fc1030084a55a81726683707e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.084259 26562 tablet_service.cc:1467] Processing CreateTablet for tablet 40c1d45639ea4980b7b376d3cd3cc1a3 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 08:05:52.069279 26691 ts_tablet_manager.cc:1398] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20240502 08:05:52.086220 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c1d45639ea4980b7b376d3cd3cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.087376 26691 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.091771 26690 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.093214 26691 tablet_bootstrap.cc:654] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.096204 26690 ts_tablet_manager.cc:1367] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.001s
I20240502 08:05:52.096354 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 99e8fc1030084a55a81726683707e539 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 08:05:52.082005 26635 tablet_service.cc:1467] Processing CreateTablet for tablet 40c1d45639ea4980b7b376d3cd3cc1a3 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 08:05:52.100454 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e8fc1030084a55a81726683707e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.104328 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c1d45639ea4980b7b376d3cd3cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:05:52.106355 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:52.111075 26690 raft_consensus.cc:348] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.111848 26690 raft_consensus.cc:374] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.112156 26690 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.112941 26690 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.115155 26559 tablet_service.cc:1467] Processing CreateTablet for tablet 99e8fc1030084a55a81726683707e539 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 08:05:52.116832 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e8fc1030084a55a81726683707e539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.103924 26703 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.120718 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:05:52.122433 26467 leader_election.cc:341] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.129542 26487 tablet_service.cc:1467] Processing CreateTablet for tablet 00f1f297bff34c27b4789483f0b3668f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 08:05:52.131299 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f1f297bff34c27b4789483f0b3668f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.132673 26690 ts_tablet_manager.cc:1398] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20240502 08:05:52.150241 26631 tablet_service.cc:1467] Processing CreateTablet for tablet 00f1f297bff34c27b4789483f0b3668f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 08:05:52.152217 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f1f297bff34c27b4789483f0b3668f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.152149 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 00f1f297bff34c27b4789483f0b3668f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 08:05:52.158300 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f1f297bff34c27b4789483f0b3668f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.159792 26486 tablet_service.cc:1467] Processing CreateTablet for tablet b4f7c38cbdaa43fa8b894d009abef926 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 08:05:52.163498 26566 tablet_service.cc:1467] Processing CreateTablet for tablet b4f7c38cbdaa43fa8b894d009abef926 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 08:05:52.164865 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f7c38cbdaa43fa8b894d009abef926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.175920 26634 tablet_service.cc:1467] Processing CreateTablet for tablet b4f7c38cbdaa43fa8b894d009abef926 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 08:05:52.182864 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:52.183408 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f7c38cbdaa43fa8b894d009abef926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.183503 26650 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.185520 26691 tablet_bootstrap.cc:492] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.186252 26691 ts_tablet_manager.cc:1367] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.099s user 0.020s sys 0.025s
I20240502 08:05:52.182139 26690 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.192862 26693 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.194358 26693 ts_tablet_manager.cc:1367] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.182s user 0.011s sys 0.004s
I20240502 08:05:52.197906 26650 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:52.198846 26693 raft_consensus.cc:348] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.199162 26467 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.202510 26690 tablet_bootstrap.cc:654] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.170261 26703 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.203850 26693 raft_consensus.cc:374] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.209179 26703 raft_consensus.cc:2781] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.210362 26693 raft_consensus.cc:729] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.210892 26703 raft_consensus.cc:686] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:52.211256 26693 consensus_queue.cc:260] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.212141 26703 consensus_queue.cc:237] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.189864 26691 raft_consensus.cc:348] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.214823 26691 raft_consensus.cc:374] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.215145 26691 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.216529 26562 tablet_service.cc:1467] Processing CreateTablet for tablet e067615c3a4148d597769a9f8a757e38 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 08:05:52.207072 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f7c38cbdaa43fa8b894d009abef926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.224454 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e067615c3a4148d597769a9f8a757e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.229524 26634 tablet_service.cc:1467] Processing CreateTablet for tablet e067615c3a4148d597769a9f8a757e38 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 08:05:52.230832 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e067615c3a4148d597769a9f8a757e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.230753 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0cb939b8c5477abcbd7707edbdc28b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 08:05:52.231513 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0cb939b8c5477abcbd7707edbdc28b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 08:05:52.232007 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0cb939b8c5477abcbd7707edbdc28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.232710 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0cb939b8c5477abcbd7707edbdc28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.232666 26485 tablet_service.cc:1467] Processing CreateTablet for tablet e067615c3a4148d597769a9f8a757e38 (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 08:05:52.234103 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e067615c3a4148d597769a9f8a757e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.243382 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0cb939b8c5477abcbd7707edbdc28b (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 08:05:52.244565 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0cb939b8c5477abcbd7707edbdc28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.252110 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc168bb1b4064b7e7801a4f94f5ff (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 08:05:52.253360 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc168bb1b4064b7e7801a4f94f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.253654 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc168bb1b4064b7e7801a4f94f5ff (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 08:05:52.254899 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc168bb1b4064b7e7801a4f94f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.258500 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc168bb1b4064b7e7801a4f94f5ff (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 08:05:52.259792 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc168bb1b4064b7e7801a4f94f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.227353 26691 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.260879 26714 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.261332 26714 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.263070 26483 tablet_service.cc:1467] Processing CreateTablet for tablet f4bbea332580449bbd811888439ee24f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 08:05:52.264403 26483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bbea332580449bbd811888439ee24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.265524 26714 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.266808 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.267520 26576 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:52.267575 26490 tablet_service.cc:1467] Processing CreateTablet for tablet d80f2fb50ae948dfb8cefba0fe36bd7f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 08:05:52.268811 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80f2fb50ae948dfb8cefba0fe36bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.268941 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:52.269538 26500 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:52.270419 26616 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:52.271842 26699 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.272218 26699 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.272553 26699 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.272778 26637 tablet_service.cc:1467] Processing CreateTablet for tablet f4bbea332580449bbd811888439ee24f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 08:05:52.274886 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bbea332580449bbd811888439ee24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.275717 26557 tablet_service.cc:1467] Processing CreateTablet for tablet f4bbea332580449bbd811888439ee24f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 08:05:52.276994 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bbea332580449bbd811888439ee24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.278364 26716 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:52.282971 26699 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.286486 26560 tablet_service.cc:1467] Processing CreateTablet for tablet d80f2fb50ae948dfb8cefba0fe36bd7f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 08:05:52.287935 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80f2fb50ae948dfb8cefba0fe36bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.289958 26640 tablet_service.cc:1467] Processing CreateTablet for tablet d80f2fb50ae948dfb8cefba0fe36bd7f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 08:05:52.291275 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80f2fb50ae948dfb8cefba0fe36bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.293448 26491 tablet_service.cc:1467] Processing CreateTablet for tablet fa9686fb37d748508274be7bfb62c56f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 08:05:52.298303 26566 tablet_service.cc:1467] Processing CreateTablet for tablet fa9686fb37d748508274be7bfb62c56f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 08:05:52.299615 26566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9686fb37d748508274be7bfb62c56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.304093 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9686fb37d748508274be7bfb62c56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.307365 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:52.307983 26500 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.309419 26697 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.309860 26697 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.311792 26697 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.313274 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.315636 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:52.316262 26501 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:52.317387 26541 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:52.318217 26697 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.318558 26697 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.318864 26697 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.318233 26714 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 08:05:52.329280 26500 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:52.329427 26732 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.325250 26697 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.337193 26697 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.334362 26732 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.334714 26615 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:52.338634 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:52.339272 26732 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.340018 26714 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.340490 26714 raft_consensus.cc:686] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:52.341179 26714 consensus_queue.cc:237] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.341655 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.342317 26575 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:52.342968 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.335541 26693 ts_tablet_manager.cc:1398] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.139s user 0.009s sys 0.002s
I20240502 08:05:52.344024 26467 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:05:52.344970 26467 leader_election.cc:341] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.344988 26693 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.346774 26699 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.347316 26732 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.347829 26732 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.351356 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:52.353401 26575 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.352360 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:52.352926 26693 tablet_bootstrap.cc:654] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.362557 26636 tablet_service.cc:1467] Processing CreateTablet for tablet fa9686fb37d748508274be7bfb62c56f (DEFAULT_TABLE table=test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 08:05:52.363648 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9686fb37d748508274be7bfb62c56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:52.370024 26500 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.360900 26732 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:52.394485 26542 leader_election.cc:341] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:52.399696 26542 leader_election.cc:341] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:52.408113 26615 consensus_peers.cc:485] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Couldn't send request to peer 8f9fe1f3763547388c4301b13a127aaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:52.395823 26691 ts_tablet_manager.cc:1398] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.209s user 0.006s sys 0.006s
I20240502 08:05:52.450891 26691 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.454934 26716 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.466531 26716 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.474910 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:52.476333 26615 leader_election.cc:341] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.476867 26575 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:52.411131 26500 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:52.486645 26733 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.487247 26733 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:52.454424 26386 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 57418.8us
I20240502 08:05:52.503037 26690 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.504029 26690 ts_tablet_manager.cc:1367] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.334s user 0.027s sys 0.016s
I20240502 08:05:52.490993 26716 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.517438 26541 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:52.518599 26697 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.524102 26691 tablet_bootstrap.cc:654] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.525246 26733 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.529557 26716 consensus_queue.cc:1035] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:52.537446 26398 catalog_manager.cc:5484] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.541709 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:52.545966 26467 leader_election.cc:341] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.546391 26690 raft_consensus.cc:348] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.547253 26690 raft_consensus.cc:374] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.565402 26690 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.555248 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.556658 26693 tablet_bootstrap.cc:492] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.567139 26693 ts_tablet_manager.cc:1367] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.222s user 0.051s sys 0.091s
I20240502 08:05:52.551201 26732 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.574113 26693 raft_consensus.cc:348] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.574873 26693 raft_consensus.cc:374] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.575119 26693 raft_consensus.cc:729] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.575685 26693 consensus_queue.cc:260] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.577421 26693 ts_tablet_manager.cc:1398] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240502 08:05:52.578207 26693 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.582195 26697 raft_consensus.cc:686] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:52.583202 26697 consensus_queue.cc:237] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.567734 26690 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:52.573385 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 118292us
I20240502 08:05:52.595718 26690 ts_tablet_manager.cc:1398] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.091s user 0.007s sys 0.000s
I20240502 08:05:52.596740 26690 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.597688 26732 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.594471 26693 tablet_bootstrap.cc:654] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.602512 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:05:52.605659 26467 leader_election.cc:341] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.605588 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:52.606374 26574 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.607596 26691 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.608947 26691 ts_tablet_manager.cc:1367] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.158s user 0.018s sys 0.020s
I20240502 08:05:52.609107 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.609809 26573 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:52.611676 26691 raft_consensus.cc:348] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.612426 26691 raft_consensus.cc:374] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.612715 26691 raft_consensus.cc:729] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.613386 26691 consensus_queue.cc:260] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.614120 26467 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.615423 26733 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:52.614748 26397 catalog_manager.cc:5484] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.616783 26691 ts_tablet_manager.cc:1398] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:05:52.617771 26691 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.621434 26398 catalog_manager.cc:5484] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.623087 26574 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:52.622246 26616 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:52.631085 26714 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.631567 26714 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.631886 26714 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.633193 26467 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:52.635727 26732 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.636240 26732 raft_consensus.cc:686] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:52.637109 26732 consensus_queue.cc:237] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.640337 26691 tablet_bootstrap.cc:654] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.630515 26690 tablet_bootstrap.cc:654] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.643411 26714 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.648970 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:52.650247 26733 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.650838 26733 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.653362 26733 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:52.656845 26615 leader_election.cc:341] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.658896 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.661914 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.664310 26467 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.665786 26732 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:52.670091 26693 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.670370 26714 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.670606 26693 ts_tablet_manager.cc:1367] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.093s user 0.022s sys 0.025s
I20240502 08:05:52.673347 26693 raft_consensus.cc:348] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.674360 26732 consensus_queue.cc:1035] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:52.676795 26397 catalog_manager.cc:5484] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.680136 26691 tablet_bootstrap.cc:492] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.680044 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:52.680626 26691 ts_tablet_manager.cc:1367] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.000s
I20240502 08:05:52.683127 26691 raft_consensus.cc:348] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.683779 26691 raft_consensus.cc:374] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.684041 26691 raft_consensus.cc:729] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.684643 26691 consensus_queue.cc:260] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.686097 26693 raft_consensus.cc:374] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.686457 26693 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.680660 26573 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.686926 26691 ts_tablet_manager.cc:1398] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 08:05:52.687849 26691 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.687345 26693 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.689975 26693 ts_tablet_manager.cc:1398] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.019s user 0.001s sys 0.004s
I20240502 08:05:52.691249 26693 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.696259 26693 tablet_bootstrap.cc:654] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.705467 26691 tablet_bootstrap.cc:654] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.716248 26690 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.736229 26690 ts_tablet_manager.cc:1367] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.140s user 0.020s sys 0.009s
I20240502 08:05:52.738587 26690 raft_consensus.cc:348] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.739146 26690 raft_consensus.cc:374] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.739400 26690 raft_consensus.cc:729] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.739917 26691 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.740375 26693 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.739959 26690 consensus_queue.cc:260] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:52.732342 26467 consensus_peers.cc:485] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa -> Peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Couldn't send request to peer 97dc9560dbbe4d1aa44b2b2edda4080f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:52.740514 26691 ts_tablet_manager.cc:1367] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.002s
I20240502 08:05:52.720604 26573 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:52.740785 26693 ts_tablet_manager.cc:1367] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.050s user 0.017s sys 0.023s
I20240502 08:05:52.742439 26616 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:52.743113 26691 raft_consensus.cc:348] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.743780 26691 raft_consensus.cc:374] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.744055 26691 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.743979 26693 raft_consensus.cc:348] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.744689 26693 raft_consensus.cc:374] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.744836 26714 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.745014 26693 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.744781 26691 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.745398 26714 raft_consensus.cc:686] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:52.745682 26693 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.746269 26714 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.748147 26693 ts_tablet_manager.cc:1398] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:52.749393 26693 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.746795 26691 ts_tablet_manager.cc:1398] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:52.754271 26690 ts_tablet_manager.cc:1398] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240502 08:05:52.754779 26693 tablet_bootstrap.cc:654] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.754859 26691 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.755280 26690 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.761417 26691 tablet_bootstrap.cc:654] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.765151 26737 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.765805 26737 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.768172 26737 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.770295 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.773059 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.773696 26650 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:52.774835 26690 tablet_bootstrap.cc:654] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.775206 26467 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:05:52.776237 26467 leader_election.cc:341] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.776751 26737 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:52.777653 26737 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:52.778264 26737 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.784328 26693 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.784962 26693 ts_tablet_manager.cc:1367] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.020s
I20240502 08:05:52.789199 26737 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.790455 26690 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.790951 26690 ts_tablet_manager.cc:1367] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.003s
I20240502 08:05:52.791651 26398 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.792912 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:52.793980 26690 raft_consensus.cc:348] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:52.794431 26467 leader_election.cc:341] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.794697 26690 raft_consensus.cc:374] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.794970 26690 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.795146 26693 raft_consensus.cc:348] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.795895 26693 raft_consensus.cc:374] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.795689 26690 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.796197 26693 raft_consensus.cc:729] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.796420 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:52.797008 26650 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.804430 26650 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:52.805255 26690 ts_tablet_manager.cc:1398] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20240502 08:05:52.805545 26467 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.806424 26703 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:52.806857 26703 raft_consensus.cc:686] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:52.807652 26703 consensus_queue.cc:237] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.809984 26693 consensus_queue.cc:260] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.811576 26693 ts_tablet_manager.cc:1398] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20240502 08:05:52.812446 26693 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.814658 26737 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.816809 26691 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.817241 26691 ts_tablet_manager.cc:1367] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.005s
I20240502 08:05:52.817376 26693 tablet_bootstrap.cc:654] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.817449 26690 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.819733 26691 raft_consensus.cc:348] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.820585 26691 raft_consensus.cc:374] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.820922 26691 raft_consensus.cc:729] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.822587 26691 consensus_queue.cc:260] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.825456 26691 ts_tablet_manager.cc:1398] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240502 08:05:52.826838 26398 catalog_manager.cc:5484] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.827911 26691 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.830474 26693 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.830886 26693 ts_tablet_manager.cc:1367] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20240502 08:05:52.833369 26691 tablet_bootstrap.cc:654] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.833281 26693 raft_consensus.cc:348] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.833700 26714 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:52.834014 26693 raft_consensus.cc:374] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.835799 26693 raft_consensus.cc:729] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.823726 26690 tablet_bootstrap.cc:654] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.836447 26693 consensus_queue.cc:260] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.838762 26693 ts_tablet_manager.cc:1398] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:05:52.839604 26693 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.845170 26693 tablet_bootstrap.cc:654] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.846833 26714 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.847352 26714 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.850059 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:52.854635 26615 leader_election.cc:341] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.858909 26714 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.859287 26740 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:52.860316 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:52.861409 26616 leader_election.cc:341] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.861763 26616 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.863328 26740 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:52.866415 26690 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.866850 26690 ts_tablet_manager.cc:1367] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.005s
I20240502 08:05:52.868038 26691 tablet_bootstrap.cc:492] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.869319 26690 raft_consensus.cc:348] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.870478 26691 ts_tablet_manager.cc:1367] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.007s
I20240502 08:05:52.870487 26690 raft_consensus.cc:374] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.870998 26690 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.871562 26693 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.871948 26693 ts_tablet_manager.cc:1367] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.006s
I20240502 08:05:52.871762 26690 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.874246 26693 raft_consensus.cc:348] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.874894 26693 raft_consensus.cc:374] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.875165 26693 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.875741 26693 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.876139 26691 raft_consensus.cc:348] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.877038 26691 raft_consensus.cc:374] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.877286 26691 raft_consensus.cc:729] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.877801 26693 ts_tablet_manager.cc:1398] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240502 08:05:52.877954 26691 consensus_queue.cc:260] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.878659 26693 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.880895 26690 ts_tablet_manager.cc:1398] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240502 08:05:52.884209 26693 tablet_bootstrap.cc:654] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.887380 26691 ts_tablet_manager.cc:1398] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240502 08:05:52.888356 26691 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.889605 26690 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.891520 26693 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.892107 26693 ts_tablet_manager.cc:1367] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 08:05:52.893786 26691 tablet_bootstrap.cc:654] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.895406 26690 tablet_bootstrap.cc:654] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.894909 26693 raft_consensus.cc:348] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.895749 26693 raft_consensus.cc:374] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.896035 26693 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.896697 26693 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.898562 26691 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.898926 26693 ts_tablet_manager.cc:1398] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:05:52.899164 26691 ts_tablet_manager.cc:1367] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240502 08:05:52.900071 26693 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.901706 26691 raft_consensus.cc:348] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.902448 26691 raft_consensus.cc:374] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.902691 26691 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.903342 26691 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.908042 26691 ts_tablet_manager.cc:1398] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240502 08:05:52.908119 26693 tablet_bootstrap.cc:654] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.908883 26691 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.914515 26691 tablet_bootstrap.cc:654] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.920991 26690 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:52.921411 26690 ts_tablet_manager.cc:1367] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.015s
I20240502 08:05:52.923874 26690 raft_consensus.cc:348] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.924419 26690 raft_consensus.cc:374] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.924508 26693 tablet_bootstrap.cc:492] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.924721 26690 raft_consensus.cc:729] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.924994 26693 ts_tablet_manager.cc:1367] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.002s
I20240502 08:05:52.925201 26691 tablet_bootstrap.cc:492] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.925621 26691 ts_tablet_manager.cc:1367] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240502 08:05:52.925330 26690 consensus_queue.cc:260] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.927616 26690 ts_tablet_manager.cc:1398] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 08:05:52.927412 26693 raft_consensus.cc:348] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.928037 26693 raft_consensus.cc:374] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.927812 26691 raft_consensus.cc:348] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.928296 26693 raft_consensus.cc:729] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.928427 26691 raft_consensus.cc:374] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.928617 26690 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:52.928680 26691 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.928957 26693 consensus_queue.cc:260] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.929387 26691 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.933833 26690 tablet_bootstrap.cc:654] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.934029 26693 ts_tablet_manager.cc:1398] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 08:05:52.934216 26703 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.934916 26693 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.934628 26703 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.936476 26703 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:52.937638 26691 ts_tablet_manager.cc:1398] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240502 08:05:52.938547 26691 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.939507 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.940004 26693 tablet_bootstrap.cc:654] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.941018 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:52.943692 26691 tablet_bootstrap.cc:654] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:52.944308 26467 leader_election.cc:341] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:52.945192 26467 leader_election.cc:341] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:52.951654 26693 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:52.952198 26693 ts_tablet_manager.cc:1367] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20240502 08:05:52.954999 26693 raft_consensus.cc:348] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.955643 26693 raft_consensus.cc:374] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.955910 26693 raft_consensus.cc:729] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.945673 26467 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.956502 26693 consensus_queue.cc:260] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.958370 26693 ts_tablet_manager.cc:1398] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:52.959141 26693 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:52.959708 26703 raft_consensus.cc:2726] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:52.955773 26691 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.961493 26691 ts_tablet_manager.cc:1367] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.003s
I20240502 08:05:52.961918 26740 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:52.962563 26740 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.963855 26691 raft_consensus.cc:348] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.964377 26693 tablet_bootstrap.cc:654] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.964531 26691 raft_consensus.cc:374] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:52.964798 26691 raft_consensus.cc:729] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:52.965039 26740 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:52.965440 26691 consensus_queue.cc:260] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:52.966938 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:52.967691 26691 ts_tablet_manager.cc:1398] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:52.968290 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:52.968905 26691 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:52.967715 26501 raft_consensus.cc:2379] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:52.968308 26500 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.973999 26615 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:52.975268 26714 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:52.982120 26691 tablet_bootstrap.cc:654] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:52.986213 26714 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:52.995365 26691 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:52.995973 26691 ts_tablet_manager.cc:1367] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.007s
I20240502 08:05:52.999531 26691 raft_consensus.cc:348] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.000550 26691 raft_consensus.cc:374] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.000938 26691 raft_consensus.cc:729] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.001868 26691 consensus_queue.cc:260] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.005389 26691 ts_tablet_manager.cc:1398] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240502 08:05:53.005395 26737 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.005983 26737 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.006942 26693 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.007398 26693 ts_tablet_manager.cc:1367] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.048s user 0.020s sys 0.024s
I20240502 08:05:53.007956 26691 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.007956 26737 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.008966 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.010282 26693 raft_consensus.cc:348] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.010967 26693 raft_consensus.cc:374] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.011219 26693 raft_consensus.cc:729] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.011755 26693 consensus_queue.cc:260] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.012504 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.013100 26650 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.013319 26691 tablet_bootstrap.cc:654] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.013679 26693 ts_tablet_manager.cc:1398] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:53.014734 26693 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.014637 26467 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.015556 26737 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.015898 26737 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.016218 26737 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.017822 26690 tablet_bootstrap.cc:492] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.018263 26690 ts_tablet_manager.cc:1367] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.090s user 0.033s sys 0.012s
W20240502 08:05:53.018415 26467 leader_election.cc:341] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.020625 26693 tablet_bootstrap.cc:654] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.020663 26690 raft_consensus.cc:348] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.021376 26697 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.021986 26697 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.024204 26697 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.026991 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.027348 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.027874 26650 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:53.028208 26500 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:53.029301 26542 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.030524 26743 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.031131 26743 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.035768 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.039357 26697 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.039899 26697 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.040361 26697 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.033843 26743 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:53.044355 26542 leader_election.cc:341] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.045305 26691 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.045679 26691 ts_tablet_manager.cc:1367] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.015s
I20240502 08:05:53.021457 26690 raft_consensus.cc:374] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.046147 26693 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.046835 26693 ts_tablet_manager.cc:1367] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.015s
I20240502 08:05:53.048048 26691 raft_consensus.cc:348] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.048766 26691 raft_consensus.cc:374] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.049036 26691 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.049732 26691 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.049988 26693 raft_consensus.cc:348] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.050971 26693 raft_consensus.cc:374] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.051338 26693 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.052110 26693 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.054774 26693 ts_tablet_manager.cc:1398] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240502 08:05:53.055948 26693 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.059387 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.060134 26691 ts_tablet_manager.cc:1398] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 08:05:53.046290 26690 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.062026 26690 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.063107 26693 tablet_bootstrap.cc:654] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.065165 26703 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.065642 26703 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.067842 26703 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.069263 26691 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.072762 26690 ts_tablet_manager.cc:1398] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20240502 08:05:53.073787 26690 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.074769 26691 tablet_bootstrap.cc:654] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:53.084647 26541 leader_election.cc:341] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.085425 26541 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.087044 26743 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.095194 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.097502 26737 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.100483 26737 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.102541 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.103008 26743 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.103439 26650 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
W20240502 08:05:53.109555 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:53.111042 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.111992 26573 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.118343 26743 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.123803 26737 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.124502 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.124442 26737 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.125211 26649 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.127067 26737 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.130714 26690 tablet_bootstrap.cc:654] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.139120 26693 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.139595 26693 ts_tablet_manager.cc:1367] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.084s user 0.029s sys 0.049s
I20240502 08:05:53.139624 26649 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.141884 26714 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.142201 26693 raft_consensus.cc:348] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.143119 26693 raft_consensus.cc:374] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.143393 26693 raft_consensus.cc:729] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.144258 26693 consensus_queue.cc:260] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.146111 26573 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.146346 26467 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.146639 26693 ts_tablet_manager.cc:1398] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 08:05:53.147050 26697 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.148627 26693 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
W20240502 08:05:53.158619 26467 leader_election.cc:341] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.160729 26467 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.159579 26737 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.161908 26703 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.162276 26737 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.162684 26703 raft_consensus.cc:686] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.162799 26737 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.163880 26703 consensus_queue.cc:237] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.172875 26714 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.173414 26714 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.176074 26714 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.177175 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.178483 26697 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.182507 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.187649 26573 raft_consensus.cc:2379] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:53.188867 26691 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.189464 26691 ts_tablet_manager.cc:1367] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.041s
I20240502 08:05:53.195183 26615 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.193976 26691 raft_consensus.cc:348] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.200719 26691 raft_consensus.cc:374] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.201175 26714 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.201503 26691 raft_consensus.cc:729] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.202813 26691 consensus_queue.cc:260] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.205718 26691 ts_tablet_manager.cc:1398] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 08:05:53.205750 26740 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.185633 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.207343 26740 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.209513 26691 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.212368 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.222612 26501 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.223222 26740 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.228952 26714 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.210395 26650 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.229591 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.231941 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.235103 26714 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.242429 26500 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.244649 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.245064 26703 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.245560 26649 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.245553 26703 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.247354 26703 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.248075 26615 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.249246 26716 raft_consensus.cc:2781] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.249589 26716 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.249918 26716 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:53.250422 26616 leader_election.cc:341] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.241916 26714 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.255983 26690 tablet_bootstrap.cc:492] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.256464 26690 ts_tablet_manager.cc:1367] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.183s user 0.020s sys 0.022s
I20240502 08:05:53.256309 26740 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.260978 26703 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.260805 26699 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.267508 26699 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.267508 26703 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.270282 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.270938 26499 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.271391 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.259783 26690 raft_consensus.cc:348] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.272089 26690 raft_consensus.cc:374] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.272383 26690 raft_consensus.cc:729] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.272475 26615 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.272933 26699 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.274865 26690 consensus_queue.cc:260] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.274618 26699 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.276813 26576 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.277504 26699 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.277932 26699 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.277803 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:53.279174 26615 leader_election.cc:341] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.279539 26693 tablet_bootstrap.cc:654] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.276535 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.281376 26733 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.281733 26733 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.283342 26733 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.285519 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.286114 26714 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.285519 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.286998 26649 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.287870 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.288501 26573 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.273903 26716 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.280412 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:53.291339 26501 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.291400 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.292009 26733 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.301234 26780 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.292814 26650 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:53.303613 26733 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.304183 26733 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.304307 26542 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.304397 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:53.294373 26397 catalog_manager.cc:5484] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.305186 26743 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.304682 26733 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.305709 26743 raft_consensus.cc:686] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:53.297556 26737 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.306649 26743 consensus_queue.cc:237] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.298761 26703 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.301915 26780 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.305299 26500 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:53.297211 26467 leader_election.cc:341] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.300480 26714 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.312031 26690 ts_tablet_manager.cc:1398] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.055s user 0.008s sys 0.002s
I20240502 08:05:53.317533 26691 tablet_bootstrap.cc:654] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.320222 26733 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.307315 26737 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.312963 26780 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.314163 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.309514 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.321980 26699 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.318423 26703 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:05:53.324085 26467 leader_election.cc:341] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.318753 26714 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.338357 26714 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.338382 26703 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:53.340443 26467 consensus_peers.cc:485] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa -> Peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Couldn't send request to peer 97dc9560dbbe4d1aa44b2b2edda4080f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:53.341264 26703 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.343122 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.337874 26650 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:53.344228 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:05:53.344708 26616 leader_election.cc:341] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.344892 26648 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.344323 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.341758 26703 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 08:05:53.332319 26690 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.317667 26716 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.354214 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.354563 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.357512 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.358215 26498 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:53.356647 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:53.359174 26499 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.354847 26649 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.360980 26714 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.348579 26790 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.362277 26542 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.362510 26790 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.364398 26500 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:53.364763 26780 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.365139 26780 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.365481 26780 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.365583 26615 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.363226 26467 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.366426 26790 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.354907 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.367987 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.368826 26716 raft_consensus.cc:2781] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.370965 26693 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.371433 26693 ts_tablet_manager.cc:1367] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.223s user 0.023s sys 0.000s
I20240502 08:05:53.366907 26703 raft_consensus.cc:2781] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.367183 26699 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.374621 26790 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 08:05:53.377177 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.377779 26573 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.367381 26648 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.385324 26501 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:53.371877 26716 raft_consensus.cc:686] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:53.387450 26693 raft_consensus.cc:348] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.387030 26716 consensus_queue.cc:237] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.387962 26693 raft_consensus.cc:374] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.388222 26693 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.388752 26693 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.375494 26703 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.393715 26499 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:53.394829 26615 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.395278 26693 ts_tablet_manager.cc:1398] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240502 08:05:53.396019 26693 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.396173 26780 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.397964 26703 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.402869 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.404124 26574 raft_consensus.cc:2379] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:53.405272 26649 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.406498 26573 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:53.407729 26693 tablet_bootstrap.cc:654] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.411185 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.411913 26649 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.401468 26740 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.413156 26740 raft_consensus.cc:686] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:53.414155 26740 consensus_queue.cc:237] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.422905 26691 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.423471 26691 ts_tablet_manager.cc:1367] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.215s user 0.020s sys 0.018s
I20240502 08:05:53.422892 26690 tablet_bootstrap.cc:654] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.421989 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:53.429595 26691 raft_consensus.cc:348] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.430341 26691 raft_consensus.cc:374] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.431489 26714 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.432341 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.432943 26574 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.424342 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.430604 26691 raft_consensus.cc:729] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
W20240502 08:05:53.436669 26615 leader_election.cc:341] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.437335 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.439419 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.441043 26574 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:53.442188 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.443305 26714 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.443794 26714 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:53.444636 26714 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.446512 26691 consensus_queue.cc:260] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.453380 26398 catalog_manager.cc:5484] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.465224 26780 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.443332 26703 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.470290 26703 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.471258 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.472016 26650 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.481284 26397 catalog_manager.cc:5484] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.485909 26649 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
W20240502 08:05:53.486294 26467 leader_election.cc:341] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:53.487177 26467 leader_election.cc:341] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.487996 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.489395 26691 ts_tablet_manager.cc:1398] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
W20240502 08:05:53.489868 26467 leader_election.cc:341] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:53.490597 26467 leader_election.cc:341] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:53.491262 26467 leader_election.cc:341] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.491685 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.492066 26703 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:53.493027 26691 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.494498 26790 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.495365 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.496845 26790 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.497305 26790 raft_consensus.cc:686] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.497262 26467 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.498266 26790 consensus_queue.cc:237] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.498716 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.498862 26650 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:53.499660 26733 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.499969 26737 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.500118 26733 raft_consensus.cc:686] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.500252 26542 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.500396 26737 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.501024 26737 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.501108 26733 consensus_queue.cc:237] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.502439 26780 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.503046 26780 raft_consensus.cc:686] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:53.503832 26780 consensus_queue.cc:237] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.505201 26690 tablet_bootstrap.cc:492] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.505831 26690 ts_tablet_manager.cc:1367] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.174s user 0.012s sys 0.006s
I20240502 08:05:53.508843 26693 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.509296 26693 ts_tablet_manager.cc:1367] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.113s user 0.014s sys 0.007s
I20240502 08:05:53.510457 26703 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.511075 26691 tablet_bootstrap.cc:654] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.508860 26690 raft_consensus.cc:348] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.512403 26690 raft_consensus.cc:374] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.512730 26690 raft_consensus.cc:729] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.515193 26693 raft_consensus.cc:348] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.516108 26693 raft_consensus.cc:374] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.516451 26693 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.517587 26690 consensus_queue.cc:260] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.520270 26693 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.523732 26690 ts_tablet_manager.cc:1398] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 08:05:53.524688 26690 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.546159 26693 ts_tablet_manager.cc:1398] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.037s user 0.005s sys 0.004s
I20240502 08:05:53.549611 26693 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.547982 26467 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.551285 26733 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.551784 26733 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.552197 26733 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.562871 26397 catalog_manager.cc:5484] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.565057 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.567654 26397 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.576042 26733 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.578647 26733 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.579342 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.580169 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:05:53.580374 26467 leader_election.cc:341] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.580170 26648 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.587482 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.563894 26737 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.596446 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.603902 26647 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.609309 26499 raft_consensus.cc:2379] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
W20240502 08:05:53.605660 26467 leader_election.cc:341] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.613174 26690 tablet_bootstrap.cc:654] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.616039 26691 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.616515 26691 ts_tablet_manager.cc:1367] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.010s
I20240502 08:05:53.655225 26690 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.646346 26693 tablet_bootstrap.cc:654] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.651388 26691 raft_consensus.cc:348] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.656286 26691 raft_consensus.cc:374] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.656613 26691 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.624625 26398 catalog_manager.cc:5484] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.657250 26691 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.628988 26737 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.659304 26691 ts_tablet_manager.cc:1398] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.043s user 0.000s sys 0.016s
I20240502 08:05:53.626142 26647 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.660284 26691 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.655980 26690 ts_tablet_manager.cc:1367] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.132s user 0.012s sys 0.012s
I20240502 08:05:53.633172 26648 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.664228 26690 raft_consensus.cc:348] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.664986 26690 raft_consensus.cc:374] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:05:53.629559 26802 log.cc:924] Time spent T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240502 08:05:53.665270 26690 raft_consensus.cc:729] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.665891 26690 consensus_queue.cc:260] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.667409 26690 ts_tablet_manager.cc:1398] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:05:53.667239 26467 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.668274 26690 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.670238 26803 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.670686 26803 raft_consensus.cc:686] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.669792 26397 catalog_manager.cc:5484] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.673213 26780 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.671233 26467 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.674324 26693 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.673605 26397 catalog_manager.cc:5484] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.671675 26803 consensus_queue.cc:237] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.674757 26693 ts_tablet_manager.cc:1367] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.011s
I20240502 08:05:53.675231 26733 raft_consensus.cc:2781] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.675915 26733 raft_consensus.cc:686] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.677222 26733 consensus_queue.cc:237] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.673656 26780 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.678448 26691 tablet_bootstrap.cc:654] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.678448 26690 tablet_bootstrap.cc:654] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.680263 26780 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.681331 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.682269 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.682473 26647 raft_consensus.cc:2379] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:53.685884 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.686682 26648 raft_consensus.cc:2379] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:53.692696 26693 raft_consensus.cc:348] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.693706 26693 raft_consensus.cc:374] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.694078 26693 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.694736 26780 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.695314 26780 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.697913 26780 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.700070 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.706782 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.708364 26693 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.720973 26690 tablet_bootstrap.cc:492] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.720917 26703 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.721459 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.722739 26693 ts_tablet_manager.cc:1398] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.048s user 0.005s sys 0.003s
I20240502 08:05:53.723412 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.723624 26693 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.724045 26690 ts_tablet_manager.cc:1367] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.056s user 0.024s sys 0.017s
I20240502 08:05:53.724166 26499 raft_consensus.cc:2379] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:53.727404 26541 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.727452 26690 raft_consensus.cc:348] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.728525 26690 raft_consensus.cc:374] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.728641 26780 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.728912 26690 raft_consensus.cc:729] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.729269 26693 tablet_bootstrap.cc:654] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.729899 26690 consensus_queue.cc:260] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.732890 26690 ts_tablet_manager.cc:1398] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:05:53.733985 26690 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.736481 26780 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.740188 26541 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.741336 26780 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.741237 26690 tablet_bootstrap.cc:654] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.748062 26780 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.750244 26733 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.750229 26780 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.751852 26691 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.752228 26691 ts_tablet_manager.cc:1367] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.092s user 0.027s sys 0.022s
I20240502 08:05:53.751492 26398 catalog_manager.cc:5484] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.753043 26398 catalog_manager.cc:5484] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.753990 26691 raft_consensus.cc:348] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.754696 26691 raft_consensus.cc:374] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.754990 26691 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.755677 26691 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.763265 26691 ts_tablet_manager.cc:1398] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 08:05:53.764185 26691 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.767588 26690 tablet_bootstrap.cc:492] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.768060 26690 ts_tablet_manager.cc:1367] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.004s
I20240502 08:05:53.770335 26690 raft_consensus.cc:348] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.771016 26690 raft_consensus.cc:374] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.771272 26690 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.773481 26693 tablet_bootstrap.cc:492] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.773911 26693 ts_tablet_manager.cc:1367] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.002s
I20240502 08:05:53.775125 26691 tablet_bootstrap.cc:654] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.778990 26697 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.780032 26693 raft_consensus.cc:348] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.780717 26693 raft_consensus.cc:374] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.780993 26693 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.781556 26693 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.783607 26693 ts_tablet_manager.cc:1398] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:05:53.784376 26693 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.787969 26733 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.789459 26693 tablet_bootstrap.cc:654] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.771831 26690 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.798182 26690 ts_tablet_manager.cc:1398] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20240502 08:05:53.799072 26690 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.801111 26733 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.801904 26733 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.804703 26733 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.800621 26573 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.809569 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.810087 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.812817 26690 tablet_bootstrap.cc:654] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.815594 26691 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.816231 26691 ts_tablet_manager.cc:1367] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.019s
W20240502 08:05:53.816936 26467 leader_election.cc:341] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:53.817896 26467 leader_election.cc:341] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:53.818316 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:53.818701 26691 raft_consensus.cc:348] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.819301 26737 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:53.819558 26691 raft_consensus.cc:374] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.819948 26691 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.820645 26691 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.822942 26691 ts_tablet_manager.cc:1398] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 08:05:53.824048 26691 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.829887 26691 tablet_bootstrap.cc:654] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.829969 26693 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.830416 26693 ts_tablet_manager.cc:1367] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.002s
I20240502 08:05:53.833176 26693 raft_consensus.cc:348] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.834009 26693 raft_consensus.cc:374] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.834396 26693 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.834540 26716 consensus_queue.cc:1035] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.835220 26693 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.837955 26690 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:53.838589 26690 ts_tablet_manager.cc:1367] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.004s
I20240502 08:05:53.841603 26576 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.841662 26690 raft_consensus.cc:348] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.851009 26690 raft_consensus.cc:374] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.851347 26690 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.853382 26733 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.854014 26733 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.856447 26690 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.861462 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.862080 26693 ts_tablet_manager.cc:1398] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.031s user 0.003s sys 0.004s
I20240502 08:05:53.862385 26575 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.863232 26693 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:53.860158 26714 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:53.867507 26714 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.859378 26733 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.872108 26716 consensus_queue.cc:1035] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.870801 26714 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:53.875322 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:53.876179 26573 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.878563 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.879441 26499 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:53.880687 26467 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.881536 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.882197 26648 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:53.882272 26733 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.882805 26733 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.883246 26733 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.884896 26616 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:53.888758 26714 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:53.891624 26690 ts_tablet_manager.cc:1398] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.053s user 0.006s sys 0.002s
I20240502 08:05:53.894373 26501 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.895550 26691 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:53.896771 26691 ts_tablet_manager.cc:1367] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.073s user 0.027s sys 0.039s
I20240502 08:05:53.897253 26690 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:53.899252 26691 raft_consensus.cc:348] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.899957 26691 raft_consensus.cc:374] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:53.900244 26691 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:53.900889 26691 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.904695 26690 tablet_bootstrap.cc:654] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.892750 26714 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:53.906634 26691 ts_tablet_manager.cc:1398] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 08:05:53.907851 26691 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:53.908942 26716 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.905714 26714 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.922348 26693 tablet_bootstrap.cc:654] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.926249 26733 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.928309 26733 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.929031 26697 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:05:53.929543 26697 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.934760 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:53.936836 26697 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:53.937260 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:53.937836 26575 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.943542 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:53.944131 26648 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:53.947507 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:53.948879 26691 tablet_bootstrap.cc:654] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:53.949009 26716 consensus_queue.cc:1035] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:53.957527 26693 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:53.958411 26693 ts_tablet_manager.cc:1367] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.095s user 0.016s sys 0.000s
I20240502 08:05:53.960609 26648 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.962544 26541 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:53.963601 26697 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:53.964378 26467 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:53.964408 26575 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:53.968248 26803 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:53.970322 26803 raft_consensus.cc:686] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:53.971446 26803 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:53.974644 26703 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:53.985553 26697 consensus_queue.cc:1035] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:54.054854 26693 raft_consensus.cc:348] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.055660 26693 raft_consensus.cc:374] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.055943 26693 raft_consensus.cc:729] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.056572 26693 consensus_queue.cc:260] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.058387 26691 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.058849 26691 ts_tablet_manager.cc:1367] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.151s user 0.050s sys 0.039s
I20240502 08:05:54.061057 26716 consensus_queue.cc:1035] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.068008 26690 tablet_bootstrap.cc:492] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.068590 26690 ts_tablet_manager.cc:1367] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.172s user 0.026s sys 0.015s
W20240502 08:05:54.068756 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 106315us
I20240502 08:05:54.069348 26691 raft_consensus.cc:348] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.070123 26691 raft_consensus.cc:374] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.070396 26691 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.070982 26690 raft_consensus.cc:348] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.071775 26690 raft_consensus.cc:374] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.072173 26690 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.072993 26690 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.070982 26691 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.075608 26690 ts_tablet_manager.cc:1398] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:05:54.076467 26690 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.077075 26691 ts_tablet_manager.cc:1398] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240502 08:05:54.076833 26714 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.077958 26691 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.080822 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.081451 26499 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.086302 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.086961 26575 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:54.089849 26467 consensus_peers.cc:485] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa -> Peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Couldn't send request to peer 97dc9560dbbe4d1aa44b2b2edda4080f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:54.091540 26733 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.094287 26693 ts_tablet_manager.cc:1398] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.135s user 0.006s sys 0.000s
I20240502 08:05:54.095522 26693 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.102006 26690 tablet_bootstrap.cc:654] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.079798 26714 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.093380 26733 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 08:05:54.115603 26399 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:54.112879 26499 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.116748 26794 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240502 08:05:54.146695 26737 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.147383 26737 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.145430 26615 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:05:54.154999 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 61846.4us
I20240502 08:05:54.155681 26836 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.156251 26836 raft_consensus.cc:686] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.157112 26836 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.161028 26575 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.149899 26737 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.187295 26693 tablet_bootstrap.cc:654] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.199995 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.211619 26714 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.212285 26714 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.214561 26697 consensus_queue.cc:1035] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.217253 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.218171 26498 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.209453 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.226467 26714 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.232687 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:54.234340 26616 leader_election.cc:341] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:54.241484 26833 log.cc:924] Time spent T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.075s user 0.016s sys 0.019s
I20240502 08:05:54.242131 26615 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.243353 26794 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.243711 26794 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.244131 26691 tablet_bootstrap.cc:654] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.244051 26794 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.252545 26690 tablet_bootstrap.cc:492] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.253008 26690 ts_tablet_manager.cc:1367] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.177s user 0.025s sys 0.027s
W20240502 08:05:54.246312 26467 leader_election.cc:341] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.260720 26690 raft_consensus.cc:348] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.261390 26690 raft_consensus.cc:374] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.261617 26690 raft_consensus.cc:729] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.262218 26690 consensus_queue.cc:260] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:54.268249 26467 leader_election.cc:341] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:54.268498 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:54.268716 26467 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.278833 26690 ts_tablet_manager.cc:1398] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20240502 08:05:54.280835 26690 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.284232 26803 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.295066 26733 consensus_queue.cc:1035] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:05:54.300284 26467 consensus_peers.cc:485] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa -> Peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Couldn't send request to peer 97dc9560dbbe4d1aa44b2b2edda4080f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:54.300696 26703 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:05:54.300963 26467 consensus_peers.cc:485] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa -> Peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Couldn't send request to peer 97dc9560dbbe4d1aa44b2b2edda4080f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:54.301177 26703 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.302852 26790 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.319442 26703 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.318059 26399 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:54.327486 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.327875 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.330523 26693 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.330978 26693 ts_tablet_manager.cc:1367] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.236s user 0.020s sys 0.004s
W20240502 08:05:54.331312 26467 leader_election.cc:341] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.335847 26693 raft_consensus.cc:348] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.336547 26693 raft_consensus.cc:374] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.336860 26693 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.338150 26691 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.338606 26691 ts_tablet_manager.cc:1367] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.261s user 0.008s sys 0.016s
I20240502 08:05:54.338656 26693 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.339679 26690 tablet_bootstrap.cc:654] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.340713 26693 ts_tablet_manager.cc:1398] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:05:54.341081 26691 raft_consensus.cc:348] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.341531 26693 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.341717 26691 raft_consensus.cc:374] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.342020 26691 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.342607 26691 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:54.343650 26467 leader_election.cc:341] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.344077 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.306243 26794 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.346231 26703 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.355507 26786 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.356034 26786 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.358423 26786 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.362227 26693 tablet_bootstrap.cc:654] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.363976 26691 ts_tablet_manager.cc:1398] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.025s user 0.002s sys 0.004s
I20240502 08:05:54.364591 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.364900 26691 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.365895 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.365955 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.364907 26794 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.369134 26500 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:54.369504 26616 leader_election.cc:341] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.370410 26690 tablet_bootstrap.cc:492] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.370864 26690 ts_tablet_manager.cc:1367] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.006s
I20240502 08:05:54.373144 26690 raft_consensus.cc:348] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.373802 26690 raft_consensus.cc:374] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.374089 26690 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.374722 26690 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.402465 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.402997 26690 ts_tablet_manager.cc:1398] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.032s user 0.011s sys 0.014s
I20240502 08:05:54.405300 26691 tablet_bootstrap.cc:654] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.405225 26541 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:54.406201 26780 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.407281 26690 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.409125 26500 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.410410 26615 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.411962 26837 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.412577 26837 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.413395 26803 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.415182 26837 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.415820 26837 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.417418 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.418201 26500 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.420310 26837 raft_consensus.cc:686] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.421290 26837 consensus_queue.cc:237] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.424768 26615 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.426525 26794 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.426870 26794 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.427186 26794 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.427721 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.428368 26573 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.438202 26836 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.438657 26836 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.440563 26690 tablet_bootstrap.cc:654] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.440644 26836 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.443434 26693 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.443902 26693 ts_tablet_manager.cc:1367] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.103s user 0.013s sys 0.020s
I20240502 08:05:54.443982 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.444746 26500 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.446190 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.446308 26693 raft_consensus.cc:348] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.447055 26693 raft_consensus.cc:374] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.447362 26693 raft_consensus.cc:729] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.448254 26615 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.448064 26693 consensus_queue.cc:260] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.449184 26837 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.449587 26837 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.449995 26837 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.450403 26693 ts_tablet_manager.cc:1398] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20240502 08:05:54.450506 26794 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.451305 26693 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.453020 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.453621 26498 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.456032 26691 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.456496 26691 ts_tablet_manager.cc:1367] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.022s
I20240502 08:05:54.458861 26691 raft_consensus.cc:348] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.459506 26691 raft_consensus.cc:374] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.459759 26691 raft_consensus.cc:729] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.460328 26691 consensus_queue.cc:260] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.460832 26803 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.462433 26498 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.462960 26691 ts_tablet_manager.cc:1398] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 08:05:54.463505 26615 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.463766 26691 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.464254 26714 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.464699 26714 raft_consensus.cc:686] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.465494 26714 consensus_queue.cc:237] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.467204 26794 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.468259 26397 catalog_manager.cc:5484] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
W20240502 08:05:54.474056 26616 leader_election.cc:341] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.475064 26693 tablet_bootstrap.cc:654] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.483222 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.483815 26574 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.489080 26691 tablet_bootstrap.cc:654] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.489064 26837 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.493379 26574 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.495620 26837 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.496742 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.503963 26498 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.507858 26693 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.508401 26693 ts_tablet_manager.cc:1367] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.000s
I20240502 08:05:54.497465 26794 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.510300 26794 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.511096 26693 raft_consensus.cc:348] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.511862 26693 raft_consensus.cc:374] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.512174 26693 raft_consensus.cc:729] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.512274 26794 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.498528 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.515162 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.522161 26690 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.513734 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.522747 26690 ts_tablet_manager.cc:1367] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.116s user 0.032s sys 0.012s
I20240502 08:05:54.523044 26500 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.524144 26615 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.524893 26794 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.512976 26693 consensus_queue.cc:260] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.525266 26794 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.525127 26690 raft_consensus.cc:348] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.525693 26794 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.525847 26690 raft_consensus.cc:374] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.526227 26690 raft_consensus.cc:729] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.527220 26693 ts_tablet_manager.cc:1398] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240502 08:05:54.528133 26693 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.513670 26498 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.529731 26397 catalog_manager.cc:5484] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:54.530959 26615 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.531886 26714 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.532359 26714 raft_consensus.cc:686] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.526805 26690 consensus_queue.cc:260] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.533146 26714 consensus_queue.cc:237] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.533779 26693 tablet_bootstrap.cc:654] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.535254 26690 ts_tablet_manager.cc:1398] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 08:05:54.539175 26690 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.539433 26691 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.539891 26691 ts_tablet_manager.cc:1367] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.004s
I20240502 08:05:54.540277 26794 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.542899 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.543467 26498 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.544214 26690 tablet_bootstrap.cc:654] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.543670 26691 raft_consensus.cc:348] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.544735 26691 raft_consensus.cc:374] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.545264 26691 raft_consensus.cc:729] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.546029 26691 consensus_queue.cc:260] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.552449 26794 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:05:54.558835 26616 leader_election.cc:341] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:54.559749 26616 leader_election.cc:341] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.561853 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:05:54.563210 26616 leader_election.cc:341] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.564174 26691 ts_tablet_manager.cc:1398] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.024s user 0.002s sys 0.004s
I20240502 08:05:54.565834 26691 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.553928 26794 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.567281 26498 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.571573 26615 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.572783 26714 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.573274 26714 raft_consensus.cc:686] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.573962 26714 consensus_queue.cc:237] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.574970 26693 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.575594 26693 ts_tablet_manager.cc:1367] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.004s
I20240502 08:05:54.570135 26794 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.577970 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.578624 26498 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:54.578800 26693 raft_consensus.cc:348] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.579633 26615 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.580567 26837 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.580911 26837 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.580997 26794 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.579802 26693 raft_consensus.cc:374] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.581288 26837 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.581499 26693 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.582499 26693 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.583925 26574 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.585197 26693 ts_tablet_manager.cc:1398] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 08:05:54.585465 26573 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.587589 26693 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.596913 26693 tablet_bootstrap.cc:654] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.606607 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.607553 26690 tablet_bootstrap.cc:492] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.608002 26690 ts_tablet_manager.cc:1367] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.069s user 0.045s sys 0.012s
I20240502 08:05:54.609025 26691 tablet_bootstrap.cc:654] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.610549 26690 raft_consensus.cc:348] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.611510 26690 raft_consensus.cc:374] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.611835 26690 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.612577 26690 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:54.611616 26616 leader_election.cc:341] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.615056 26690 ts_tablet_manager.cc:1398] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:05:54.619167 26690 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.620739 26693 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.620800 26837 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.623765 26693 ts_tablet_manager.cc:1367] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.001s
I20240502 08:05:54.625766 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:54.626402 26498 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.626209 26693 raft_consensus.cc:348] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.626947 26693 raft_consensus.cc:374] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.627087 26690 tablet_bootstrap.cc:654] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.627261 26693 raft_consensus.cc:729] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.628589 26691 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.629031 26691 ts_tablet_manager.cc:1367] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.000s
I20240502 08:05:54.631918 26691 raft_consensus.cc:348] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.632808 26691 raft_consensus.cc:374] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.633147 26691 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.634075 26693 consensus_queue.cc:260] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.622749 26397 catalog_manager.cc:5484] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:54.634088 26691 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.636173 26693 ts_tablet_manager.cc:1398] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:05:54.639899 26693 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.640120 26691 ts_tablet_manager.cc:1398] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240502 08:05:54.642012 26691 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.636046 26498 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:54.649291 26837 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:54.649662 26615 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.652230 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.653263 26690 tablet_bootstrap.cc:492] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.653656 26690 ts_tablet_manager.cc:1367] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.007s
W20240502 08:05:54.653817 26616 leader_election.cc:341] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.655139 26691 tablet_bootstrap.cc:654] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.657640 26690 raft_consensus.cc:348] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.657476 26398 catalog_manager.cc:5484] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:54.658375 26690 raft_consensus.cc:374] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.658713 26690 raft_consensus.cc:729] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.660610 26690 consensus_queue.cc:260] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.662799 26690 ts_tablet_manager.cc:1398] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 08:05:54.663962 26690 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.666167 26837 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.666774 26837 raft_consensus.cc:686] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:54.669047 26690 tablet_bootstrap.cc:654] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.667603 26837 consensus_queue.cc:237] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.671314 26693 tablet_bootstrap.cc:654] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.672824 26573 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.678162 26690 tablet_bootstrap.cc:492] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.678757 26690 ts_tablet_manager.cc:1367] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240502 08:05:54.683684 26690 raft_consensus.cc:348] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.684636 26690 raft_consensus.cc:374] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.684978 26690 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.686076 26693 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.685777 26690 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.686825 26693 ts_tablet_manager.cc:1367] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.004s
I20240502 08:05:54.688645 26690 ts_tablet_manager.cc:1398] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:05:54.689786 26693 raft_consensus.cc:348] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.690593 26693 raft_consensus.cc:374] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.690892 26693 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.691499 26693 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.697283 26690 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.697892 26803 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.698508 26803 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.693542 26693 ts_tablet_manager.cc:1398] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:54.700680 26794 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:54.700476 26803 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.702458 26693 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.705502 26691 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.706064 26691 ts_tablet_manager.cc:1367] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.019s
I20240502 08:05:54.706379 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.707023 26647 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:54.709316 26690 tablet_bootstrap.cc:654] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.709489 26691 raft_consensus.cc:348] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.714871 26691 raft_consensus.cc:374] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.715190 26691 raft_consensus.cc:729] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.715880 26691 consensus_queue.cc:260] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.718243 26691 ts_tablet_manager.cc:1398] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 08:05:54.718657 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.719089 26691 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.719318 26574 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:54.720325 26690 tablet_bootstrap.cc:492] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.721269 26690 ts_tablet_manager.cc:1367] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20240502 08:05:54.721194 26733 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.713976 26693 tablet_bootstrap.cc:654] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.721796 26733 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.725142 26691 tablet_bootstrap.cc:654] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.721195 26467 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:54.738806 26733 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.739713 26733 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.740224 26733 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.742213 26733 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.744663 26786 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.745307 26786 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.746578 26690 raft_consensus.cc:348] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.747839 26786 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.749624 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.752954 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.756719 26690 raft_consensus.cc:374] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.757210 26690 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.758594 26690 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.759670 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.760691 26693 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.753937 26733 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.760691 26690 ts_tablet_manager.cc:1398] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240502 08:05:54.762207 26690 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.762631 26733 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.735241 26837 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20240502 08:05:54.713459 26396 catalog_manager.cc:5484] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:54.767550 26690 tablet_bootstrap.cc:654] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:54.773069 26467 leader_election.cc:341] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:54.774194 26467 leader_election.cc:341] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.778934 26467 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.775139 26803 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.780046 26803 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.780752 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.781368 26733 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.781530 26500 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:54.776427 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:54.763195 26693 ts_tablet_manager.cc:1367] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.061s user 0.024s sys 0.019s
I20240502 08:05:54.782281 26650 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.783288 26690 tablet_bootstrap.cc:492] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.783776 26690 ts_tablet_manager.cc:1367] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.004s
I20240502 08:05:54.786089 26690 raft_consensus.cc:348] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.786888 26690 raft_consensus.cc:374] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.787199 26690 raft_consensus.cc:729] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.787936 26690 consensus_queue.cc:260] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.789703 26690 ts_tablet_manager.cc:1398] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240502 08:05:54.790562 26786 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.791103 26786 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.793022 26786 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.794152 26790 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.794435 26693 raft_consensus.cc:348] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.794667 26790 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.796120 26690 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:54.796453 26650 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:54.796994 26790 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.798393 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.798655 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:54.799415 26574 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.795238 26693 raft_consensus.cc:374] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.800014 26693 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.801470 26693 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.803333 26693 ts_tablet_manager.cc:1398] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240502 08:05:54.804039 26693 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.807996 26690 tablet_bootstrap.cc:654] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.808051 26693 tablet_bootstrap.cc:654] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.809506 26803 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.817970 26691 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.819526 26691 ts_tablet_manager.cc:1367] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.101s user 0.041s sys 0.039s
I20240502 08:05:54.820072 26693 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:54.820503 26693 ts_tablet_manager.cc:1367] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240502 08:05:54.821682 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.822077 26691 raft_consensus.cc:348] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.822732 26691 raft_consensus.cc:374] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.822835 26693 raft_consensus.cc:348] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.824652 26693 raft_consensus.cc:374] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.824651 26691 raft_consensus.cc:729] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.825102 26693 raft_consensus.cc:729] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.825614 26691 consensus_queue.cc:260] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.825697 26693 consensus_queue.cc:260] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.827698 26693 ts_tablet_manager.cc:1398] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:05:54.827782 26691 ts_tablet_manager.cc:1398] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:05:54.829174 26693 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:54.829243 26691 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.830585 26541 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:54.831634 26786 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.834828 26691 tablet_bootstrap.cc:654] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.838279 26786 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.839008 26693 tablet_bootstrap.cc:654] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:54.840858 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:54.843885 26574 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:54.853271 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:54.854925 26467 leader_election.cc:341] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.855346 26690 tablet_bootstrap.cc:492] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:54.856012 26690 ts_tablet_manager.cc:1367] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.009s
I20240502 08:05:54.855926 26467 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:54.854641 26647 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:54.855327 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.857194 26790 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:54.861012 26650 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:54.862087 26790 raft_consensus.cc:686] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:54.862694 26691 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:54.863129 26691 ts_tablet_manager.cc:1367] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.011s
I20240502 08:05:54.863015 26790 consensus_queue.cc:237] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.866912 26691 raft_consensus.cc:348] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.867666 26691 raft_consensus.cc:374] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.867960 26691 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.869275 26467 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:54.870114 26467 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:54.868718 26691 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:54.872117 26467 leader_election.cc:341] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:54.860236 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:54.873253 26498 raft_consensus.cc:2379] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:54.873446 26733 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.873728 26733 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.873979 26733 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.874315 26541 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:54.875381 26786 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.872222 26803 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:54.878479 26803 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:54.878798 26803 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:54.881465 26786 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:54.888226 26690 raft_consensus.cc:348] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.889109 26690 raft_consensus.cc:374] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:54.889532 26690 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:54.892536 26786 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.893023 26786 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.894133 26690 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.894984 26780 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.895252 26786 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.895439 26780 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.896867 26691 ts_tablet_manager.cc:1398] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20240502 08:05:54.902572 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:54.903939 26733 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.911729 26691 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:54.913494 26803 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:54.929068 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:05:54.932355 26467 leader_election.cc:341] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:55.112776 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 180438us
I20240502 08:05:54.991071 26690 ts_tablet_manager.cc:1398] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.135s user 0.011s sys 0.000s
I20240502 08:05:55.116483 26690 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:55.123714 26693 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:55.124383 26693 ts_tablet_manager.cc:1367] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.295s user 0.012s sys 0.016s
W20240502 08:05:55.027289 26882 log.cc:924] Time spent T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240502 08:05:54.933766 26780 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.028872 26786 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:54.963660 26733 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:54.931028 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.103281 26703 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.033496 26803 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.141187 26887 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.148145 26499 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:55.135752 26786 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.141609 26703 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.164345 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:55.165247 26575 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.170091 26691 tablet_bootstrap.cc:654] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.151212 26887 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.193413 26690 tablet_bootstrap.cc:654] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.194185 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:55.184195 26733 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.189515 26786 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.194779 26650 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.230194 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.231073 26499 raft_consensus.cc:2379] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:55.233855 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:55.236694 26647 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.238256 26575 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:55.238435 26541 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:55.235870 26693 raft_consensus.cc:348] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.241179 26693 raft_consensus.cc:374] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.241394 26693 raft_consensus.cc:729] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.244161 26887 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.246151 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.215806 26733 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.261639 26498 raft_consensus.cc:2379] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:55.275399 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.284451 26733 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.297964 26397 catalog_manager.cc:5484] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.350924 26467 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:55.350610 26693 consensus_queue.cc:260] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.276307 26780 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.295920 26703 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.360158 26541 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:55.362274 26898 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.365514 26693 ts_tablet_manager.cc:1398] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.241s user 0.015s sys 0.016s
I20240502 08:05:55.366544 26693 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:55.367694 26691 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:55.368104 26691 ts_tablet_manager.cc:1367] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.456s user 0.026s sys 0.009s
I20240502 08:05:55.341665 26650 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:55.370793 26690 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:55.370751 26837 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.371399 26690 ts_tablet_manager.cc:1367] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.255s user 0.014s sys 0.006s
I20240502 08:05:55.371387 26837 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.364959 26716 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 08:05:55.382452 26467 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.373708 26837 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:55.354142 26890 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:55.402539 26690 raft_consensus.cc:348] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.403271 26690 raft_consensus.cc:374] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.403573 26690 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.404212 26690 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.400404 26703 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:55.402181 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.406463 26890 raft_consensus.cc:686] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:55.420033 26690 ts_tablet_manager.cc:1398] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20240502 08:05:55.426854 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.429111 26693 tablet_bootstrap.cc:654] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.415098 26703 raft_consensus.cc:686] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:55.430516 26573 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:55.441877 26467 leader_election.cc:341] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.443185 26890 consensus_queue.cc:237] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.445994 26889 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.440735 26542 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:55.447196 26887 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 08:05:55.453882 26889 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.451949 26703 consensus_queue.cc:237] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.462205 26889 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.457789 26647 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:55.465699 26690 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:55.467728 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.469595 26691 raft_consensus.cc:348] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.475497 26691 raft_consensus.cc:374] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.475885 26691 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.483307 26837 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.484398 26837 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.492694 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.493016 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.505508 26896 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:05:55.493928 26837 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:55.511047 26892 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 08:05:55.513089 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.514936 26899 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:05:55.515888 26467 leader_election.cc:341] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.517671 26904 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.517807 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.518489 26501 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:55.518234 26904 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.524345 26576 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
W20240502 08:05:55.537060 26467 leader_election.cc:341] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.533461 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.539034 26643 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:55.540649 26693 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:55.541119 26467 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:55.542640 26703 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:55.542999 26703 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:55.543365 26703 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.520180 26837 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240502 08:05:55.541119 26693 ts_tablet_manager.cc:1367] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.175s user 0.014s sys 0.016s
I20240502 08:05:55.548879 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.553822 26780 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.557292 26693 raft_consensus.cc:348] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.568717 26693 raft_consensus.cc:374] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.569012 26693 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.569658 26693 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:55.558303 26467 leader_election.cc:341] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.560225 26690 tablet_bootstrap.cc:654] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.560735 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.562234 26904 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.563130 26716 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:55.565388 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.565018 26615 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:55.570719 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.576148 26693 ts_tablet_manager.cc:1398] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20240502 08:05:55.576804 26693 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
W20240502 08:05:55.575475 26542 leader_election.cc:341] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.577972 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:55.579197 26887 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:55.579566 26887 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:55.579891 26887 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:55.580993 26467 leader_election.cc:341] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.581286 26467 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.583268 26716 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:55.583707 26716 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:55.584055 26716 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:55.588955 26467 leader_election.cc:341] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.578635 26886 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.594616 26885 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.595970 26703 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.600678 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.607246 26691 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.614948 26894 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.615485 26894 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.606793 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.628111 26691 ts_tablet_manager.cc:1398] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.260s user 0.013s sys 0.013s
I20240502 08:05:55.629396 26691 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:55.637079 26894 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:55.622777 26886 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240502 08:05:55.649744 26837 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:55.686450 26887 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.687867 26690 tablet_bootstrap.cc:492] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:55.688530 26690 ts_tablet_manager.cc:1367] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.223s user 0.017s sys 0.017s
I20240502 08:05:55.696128 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:55.697175 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:55.697528 26467 leader_election.cc:341] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.698105 26497 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:55.713055 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:55.694167 26703 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.730770 26691 tablet_bootstrap.cc:654] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.734555 26693 tablet_bootstrap.cc:654] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:55.719595 26615 leader_election.cc:341] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:55.742477 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:55.745030 26890 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:55.745648 26890 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.745074 26836 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:05:55.747838 26890 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.748384 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.745673 26893 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 08:05:55.745815 26892 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 08:05:55.750495 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.752841 26887 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:55.771591 26690 raft_consensus.cc:348] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.773610 26690 raft_consensus.cc:374] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.773931 26690 raft_consensus.cc:729] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.775676 26690 consensus_queue.cc:260] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.770939 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:55.779330 26569 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.779253 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:55.775799 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:55.781430 26499 raft_consensus.cc:2379] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:55.777179 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:55.782395 26541 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:55.786343 26690 ts_tablet_manager.cc:1398] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.097s user 0.007s sys 0.002s
I20240502 08:05:55.787190 26690 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
W20240502 08:05:55.787776 26467 leader_election.cc:341] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.782802 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:05:55.790613 26467 leader_election.cc:341] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.796614 26837 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:05:55.799397 26616 leader_election.cc:341] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.799623 26716 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.800825 26642 raft_consensus.cc:2379] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:55.801599 26716 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:05:55.803153 26467 leader_election.cc:341] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.777675 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:55.803717 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.804663 26703 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.808058 26467 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.809602 26885 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.809556 26892 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:55.811496 26892 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:55.811898 26892 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:55.826421 26569 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:55.829524 26904 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.830561 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:55.831313 26885 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:55.831532 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:55.831787 26885 raft_consensus.cc:686] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:55.832397 26497 raft_consensus.cc:2379] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:55.832679 26885 consensus_queue.cc:237] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.840160 26690 tablet_bootstrap.cc:654] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.848863 26892 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:55.856721 26616 leader_election.cc:341] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.857102 26693 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:55.857779 26693 ts_tablet_manager.cc:1367] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.281s user 0.014s sys 0.017s
I20240502 08:05:55.860006 26693 raft_consensus.cc:348] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.860523 26693 raft_consensus.cc:374] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.860734 26693 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
W20240502 08:05:55.860755 26616 leader_election.cc:341] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:55.861438 26616 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:55.861600 26693 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.862450 26837 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:55.863476 26693 ts_tablet_manager.cc:1398] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:05:55.864362 26693 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
W20240502 08:05:55.886410 26913 log.cc:924] Time spent T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.138s user 0.000s sys 0.002s
I20240502 08:05:55.898914 26691 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:55.899576 26691 ts_tablet_manager.cc:1367] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.270s user 0.018s sys 0.021s
I20240502 08:05:55.913548 26691 raft_consensus.cc:348] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.914458 26691 raft_consensus.cc:374] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.914795 26691 raft_consensus.cc:729] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.915549 26691 consensus_queue.cc:260] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:55.900214 26924 log.cc:924] Time spent T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240502 08:05:55.924194 26691 ts_tablet_manager.cc:1398] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20240502 08:05:55.925400 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:55.925190 26691 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
W20240502 08:05:55.981143 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 51705.8us
W20240502 08:05:55.984618 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 57172.5us
I20240502 08:05:55.985786 26703 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:55.987108 26690 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:55.987562 26690 ts_tablet_manager.cc:1367] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.200s user 0.010s sys 0.009s
I20240502 08:05:55.988371 26693 tablet_bootstrap.cc:654] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:55.991030 26690 raft_consensus.cc:348] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.991736 26690 raft_consensus.cc:374] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:55.992043 26690 raft_consensus.cc:729] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:55.992736 26892 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:55.992653 26690 consensus_queue.cc:260] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:55.994930 26690 ts_tablet_manager.cc:1398] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:05:55.995865 26690 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:55.996251 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:55.996855 26498 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.009181 26885 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.009434 26397 catalog_manager.cc:5484] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:56.015959 26397 catalog_manager.cc:5484] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:56.006846 26616 leader_election.cc:341] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.014101 26885 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.033370 26885 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.035544 26889 consensus_queue.cc:1035] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:56.031486 26616 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.038630 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.038724 26892 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.038400 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.040560 26885 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:05:56.049789 26467 leader_election.cc:341] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.079957 26780 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.090298 26780 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.092247 26780 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.131136 26691 tablet_bootstrap.cc:654] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:56.132324 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 80055.8us
I20240502 08:05:56.109972 26498 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:56.139652 26690 tablet_bootstrap.cc:654] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.143903 26693 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.144604 26693 ts_tablet_manager.cc:1367] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.280s user 0.019s sys 0.008s
I20240502 08:05:56.093904 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:56.149178 26467 leader_election.cc:341] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.150260 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.150421 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.149662 26467 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.161370 26703 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.160003 26892 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.165011 26892 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:56.166221 26892 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:56.186515 26921 log.cc:924] Time spent T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.094s user 0.004s sys 0.012s
I20240502 08:05:56.196970 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:56.198645 26570 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.232705 26780 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.233239 26780 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.235872 26780 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:56.242501 26616 leader_election.cc:341] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.249075 26693 raft_consensus.cc:348] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.249809 26693 raft_consensus.cc:374] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.250103 26693 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.250663 26693 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.253660 26693 ts_tablet_manager.cc:1398] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.109s user 0.005s sys 0.006s
I20240502 08:05:56.254500 26693 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.256989 26542 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:56.263353 26690 tablet_bootstrap.cc:492] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.263880 26690 ts_tablet_manager.cc:1367] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.268s user 0.023s sys 0.030s
I20240502 08:05:56.266286 26690 raft_consensus.cc:348] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.267100 26690 raft_consensus.cc:374] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.267107 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.267697 26690 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.268481 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.269299 26496 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
W20240502 08:05:56.270503 26467 consensus_peers.cc:485] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa -> Peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Couldn't send request to peer 755f35bc66a841f382d1c4df6c6f6b64. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:56.270956 26542 leader_election.cc:341] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.271433 26690 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.273473 26690 ts_tablet_manager.cc:1398] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:05:56.274502 26690 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.275215 26541 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:56.277374 26904 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.277760 26904 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.278164 26904 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.279152 26693 tablet_bootstrap.cc:654] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.280233 26690 tablet_bootstrap.cc:654] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.284305 26691 tablet_bootstrap.cc:492] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.284706 26691 ts_tablet_manager.cc:1367] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.360s user 0.009s sys 0.017s
I20240502 08:05:56.286885 26691 raft_consensus.cc:348] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.287515 26691 raft_consensus.cc:374] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.287771 26691 raft_consensus.cc:729] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.285404 26904 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.288511 26690 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.288497 26691 consensus_queue.cc:260] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.289042 26690 ts_tablet_manager.cc:1367] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20240502 08:05:56.290383 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:56.290984 26642 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.292534 26693 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.293032 26693 ts_tablet_manager.cc:1367] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.001s
I20240502 08:05:56.294188 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.294761 26496 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.295629 26691 ts_tablet_manager.cc:1398] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 08:05:56.298552 26691 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.298990 26904 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.297209 26693 raft_consensus.cc:348] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.301239 26693 raft_consensus.cc:374] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.301558 26693 raft_consensus.cc:729] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.302943 26690 raft_consensus.cc:348] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.302819 26693 consensus_queue.cc:260] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.303658 26690 raft_consensus.cc:374] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.303972 26690 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.304050 26780 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.306357 26887 consensus_queue.cc:1035] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:56.308323 26642 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:56.301115 26496 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:56.309374 26542 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.311370 26780 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.311836 26780 raft_consensus.cc:686] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:56.312561 26691 tablet_bootstrap.cc:654] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.312666 26780 consensus_queue.cc:237] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.315992 26693 ts_tablet_manager.cc:1398] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20240502 08:05:56.315250 26690 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.316879 26693 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.317920 26690 ts_tablet_manager.cc:1398] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20240502 08:05:56.318586 26690 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.323148 26693 tablet_bootstrap.cc:654] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.326368 26690 tablet_bootstrap.cc:654] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.334870 26691 tablet_bootstrap.cc:492] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.335311 26691 ts_tablet_manager.cc:1367] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.004s
I20240502 08:05:56.336687 26693 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.337095 26693 ts_tablet_manager.cc:1367] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240502 08:05:56.338587 26691 raft_consensus.cc:348] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.339320 26691 raft_consensus.cc:374] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.339570 26691 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.342211 26887 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:56.349838 26892 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.349539 26691 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.350353 26892 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.351804 26691 ts_tablet_manager.cc:1398] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 08:05:56.352317 26892 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.352679 26691 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.354072 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.354166 26397 catalog_manager.cc:5484] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:56.355628 26616 leader_election.cc:341] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.358592 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.359239 26496 raft_consensus.cc:2454] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:56.360868 26615 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.362004 26716 raft_consensus.cc:2781] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.362493 26716 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.362897 26716 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.369706 26691 tablet_bootstrap.cc:654] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.376726 26716 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.379139 26396 catalog_manager.cc:5484] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:56.390771 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:56.387440 26693 raft_consensus.cc:348] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.392099 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.392673 26496 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.391494 26693 raft_consensus.cc:374] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.393316 26693 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.394099 26693 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.394845 26716 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:05:56.397601 26616 leader_election.cc:341] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.403008 26693 ts_tablet_manager.cc:1398] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.066s user 0.007s sys 0.001s
I20240502 08:05:56.403832 26496 raft_consensus.cc:2454] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:56.404100 26693 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.402156 26690 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.404784 26690 ts_tablet_manager.cc:1367] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.086s user 0.029s sys 0.041s
I20240502 08:05:56.405061 26615 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.406239 26716 raft_consensus.cc:2781] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.406906 26716 raft_consensus.cc:686] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:56.407370 26690 raft_consensus.cc:348] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.408147 26690 raft_consensus.cc:374] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.407711 26716 consensus_queue.cc:237] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.408428 26690 raft_consensus.cc:729] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.409885 26690 consensus_queue.cc:260] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.412273 26690 ts_tablet_manager.cc:1398] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:05:56.413105 26690 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.418442 26690 tablet_bootstrap.cc:654] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.421778 26693 tablet_bootstrap.cc:654] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.423996 26691 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.424670 26691 ts_tablet_manager.cc:1367] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.000s
I20240502 08:05:56.427376 26691 raft_consensus.cc:348] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.428316 26691 raft_consensus.cc:374] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.428651 26691 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.432682 26691 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.435058 26691 ts_tablet_manager.cc:1398] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 08:05:56.436187 26691 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.438771 26399 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:56.441355 26691 tablet_bootstrap.cc:654] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.442802 26885 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.443480 26885 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.445545 26885 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.446647 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.446784 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.449209 26690 tablet_bootstrap.cc:492] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.447270 26780 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.449669 26690 ts_tablet_manager.cc:1367] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.015s
I20240502 08:05:56.449753 26780 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.451813 26780 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.452481 26690 raft_consensus.cc:348] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:56.452872 26467 leader_election.cc:341] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.453192 26690 raft_consensus.cc:374] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.453451 26690 raft_consensus.cc:729] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.453861 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.454079 26690 consensus_queue.cc:260] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.455145 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.455895 26496 raft_consensus.cc:2379] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:56.456544 26690 ts_tablet_manager.cc:1398] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:05:56.457444 26690 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.459856 26693 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.462472 26690 tablet_bootstrap.cc:654] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:56.454164 26467 leader_election.cc:341] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.474385 26467 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.475677 26541 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:56.476402 26885 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.476766 26780 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.478296 26396 catalog_manager.cc:5484] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:56.480674 26693 ts_tablet_manager.cc:1367] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.001s
I20240502 08:05:56.486227 26693 raft_consensus.cc:348] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.490307 26693 raft_consensus.cc:374] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.491400 26693 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.492376 26693 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.494112 26885 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.494753 26885 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.497370 26885 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.497953 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.499123 26693 ts_tablet_manager.cc:1398] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.017s user 0.002s sys 0.006s
W20240502 08:05:56.500079 26467 leader_election.cc:341] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.500380 26693 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.503929 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.505520 26642 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:56.506948 26467 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.507591 26780 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.508596 26885 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.509343 26885 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.509922 26885 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.528506 26691 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.534402 26691 ts_tablet_manager.cc:1367] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.098s user 0.018s sys 0.034s
I20240502 08:05:56.536224 26885 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.537597 26693 tablet_bootstrap.cc:654] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.537770 26892 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.538260 26892 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.540248 26892 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.537612 26691 raft_consensus.cc:348] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.540946 26691 raft_consensus.cc:374] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.541052 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.541795 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.542510 26690 tablet_bootstrap.cc:492] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.541245 26691 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.543032 26616 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.543134 26691 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.543828 26892 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.544382 26690 ts_tablet_manager.cc:1367] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.007s
I20240502 08:05:56.545432 26691 ts_tablet_manager.cc:1398] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20240502 08:05:56.546836 26690 raft_consensus.cc:348] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.547495 26690 raft_consensus.cc:374] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.547611 26693 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.547785 26690 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.548072 26693 ts_tablet_manager.cc:1367] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.000s
I20240502 08:05:56.548379 26690 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.550371 26690 ts_tablet_manager.cc:1398] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:05:56.551318 26885 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.551993 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:56.552492 26690 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.552578 26571 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.558383 26690 tablet_bootstrap.cc:654] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.558290 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:56.558987 26642 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.559319 26571 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:56.563170 26690 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.564544 26690 ts_tablet_manager.cc:1367] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 08:05:56.564966 26642 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:56.566614 26467 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.566706 26690 raft_consensus.cc:348] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.567276 26690 raft_consensus.cc:374] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.567333 26885 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.567611 26690 raft_consensus.cc:729] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.563221 26691 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.563586 26693 raft_consensus.cc:348] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.567836 26885 raft_consensus.cc:686] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:56.570780 26885 consensus_queue.cc:237] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.573390 26691 tablet_bootstrap.cc:654] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.574622 26693 raft_consensus.cc:374] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.574949 26693 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.576700 26693 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.579759 26780 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.580185 26780 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.581146 26693 ts_tablet_manager.cc:1398] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20240502 08:05:56.581903 26780 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.582389 26693 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.585055 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.585825 26498 raft_consensus.cc:2379] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:56.585006 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.586392 26690 consensus_queue.cc:260] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.587363 26642 raft_consensus.cc:2379] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:56.588346 26542 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:56.589423 26780 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.594918 26780 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.589457 26690 ts_tablet_manager.cc:1398] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20240502 08:05:56.597040 26690 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.598424 26693 tablet_bootstrap.cc:654] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.602566 26690 tablet_bootstrap.cc:654] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.604341 26398 catalog_manager.cc:5484] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:56.608541 26691 tablet_bootstrap.cc:492] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.609122 26691 ts_tablet_manager.cc:1367] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.004s
I20240502 08:05:56.612291 26691 raft_consensus.cc:348] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.613250 26691 raft_consensus.cc:374] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.613580 26691 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.614535 26691 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.617195 26691 ts_tablet_manager.cc:1398] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:05:56.618283 26691 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.631575 26690 tablet_bootstrap.cc:492] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.632201 26690 ts_tablet_manager.cc:1367] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.005s
I20240502 08:05:56.635493 26690 raft_consensus.cc:348] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.636466 26690 raft_consensus.cc:374] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.636811 26690 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.637620 26690 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.638352 26837 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.638880 26837 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.642352 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.642731 26691 tablet_bootstrap.cc:654] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.643056 26498 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:56.644361 26615 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.640481 26690 ts_tablet_manager.cc:1398] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 08:05:56.645988 26716 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.647025 26716 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.647336 26716 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.647644 26690 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.653237 26837 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.654769 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.655474 26576 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:56.658926 26690 tablet_bootstrap.cc:654] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.661273 26693 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.661726 26693 ts_tablet_manager.cc:1367] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.080s user 0.041s sys 0.026s
I20240502 08:05:56.663507 26691 tablet_bootstrap.cc:492] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.663923 26691 ts_tablet_manager.cc:1367] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.000s
I20240502 08:05:56.664144 26693 raft_consensus.cc:348] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.664829 26693 raft_consensus.cc:374] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.665066 26693 raft_consensus.cc:729] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.665627 26693 consensus_queue.cc:260] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.666180 26691 raft_consensus.cc:348] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.666831 26691 raft_consensus.cc:374] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.667083 26691 raft_consensus.cc:729] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.668033 26693 ts_tablet_manager.cc:1398] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 08:05:56.667793 26691 consensus_queue.cc:260] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.668874 26693 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.669766 26691 ts_tablet_manager.cc:1398] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:05:56.670634 26691 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.674779 26693 tablet_bootstrap.cc:654] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.675642 26691 tablet_bootstrap.cc:654] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.676038 26716 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.678282 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.678752 26500 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.679602 26693 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.679948 26693 ts_tablet_manager.cc:1367] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 08:05:56.681150 26691 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.681543 26691 ts_tablet_manager.cc:1367] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 08:05:56.682757 26693 raft_consensus.cc:348] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.683436 26693 raft_consensus.cc:374] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.683686 26693 raft_consensus.cc:729] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.683930 26691 raft_consensus.cc:348] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.684593 26691 raft_consensus.cc:374] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.684252 26693 consensus_queue.cc:260] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.684839 26691 raft_consensus.cc:729] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.686213 26892 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:56.686669 26693 ts_tablet_manager.cc:1398] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 08:05:56.686800 26690 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.688071 26690 ts_tablet_manager.cc:1367] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.000s
I20240502 08:05:56.688437 26693 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:56.690506 26690 raft_consensus.cc:348] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.691166 26690 raft_consensus.cc:374] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.691376 26690 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.687402 26691 consensus_queue.cc:260] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.695653 26500 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:56.695600 26690 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.696108 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:56.696848 26571 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.697368 26615 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.697845 26690 ts_tablet_manager.cc:1398] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:05:56.699331 26690 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.698324 26889 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.701287 26889 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.702764 26716 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.706856 26691 ts_tablet_manager.cc:1398] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.025s user 0.004s sys 0.003s
I20240502 08:05:56.707190 26889 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.707824 26691 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:56.708876 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.711179 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:56.712379 26467 leader_election.cc:341] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.713049 26571 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
W20240502 08:05:56.713362 26467 leader_election.cc:341] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.714387 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.715706 26885 raft_consensus.cc:2726] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.712474 26892 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.716452 26892 raft_consensus.cc:686] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:56.707217 26693 tablet_bootstrap.cc:654] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.717257 26892 consensus_queue.cc:237] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.718499 26690 tablet_bootstrap.cc:654] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.724874 26690 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:56.725333 26690 ts_tablet_manager.cc:1367] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240502 08:05:56.727134 26837 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.727583 26837 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.735292 26885 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.734797 26780 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.728276 26571 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.745929 26904 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.782029 26885 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.782163 26780 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.773339 26976 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240502 08:05:56.788041 26780 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.782728 26904 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.789484 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.786765 26889 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.790210 26496 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:56.790243 26889 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.790771 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.791908 26615 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.789798 26904 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.792769 26892 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.792883 26889 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.793182 26892 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.727552 26690 raft_consensus.cc:348] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.793537 26837 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.793642 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.794466 26642 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.733929 26691 tablet_bootstrap.cc:654] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.795586 26542 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.796252 26885 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.802063 26899 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.802697 26899 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.808641 26904 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.808826 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.808976 26899 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.809041 26904 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.809612 26904 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:56.810137 26616 leader_election.cc:341] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.793546 26892 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.815112 26976 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.815572 26976 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.808512 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.816157 26904 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.760908 26920 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.818555 26920 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.818998 26976 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.819514 26645 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.819579 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.820353 26642 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.793675 26690 raft_consensus.cc:374] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.822170 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:56.823083 26690 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.823202 26644 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.823840 26690 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.826162 26690 ts_tablet_manager.cc:1398] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.101s user 0.008s sys 0.000s
I20240502 08:05:56.827133 26542 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.827404 26690 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:56.828403 26542 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.829382 26899 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.829710 26644 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:56.830269 26976 raft_consensus.cc:2781] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.829787 26899 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.830664 26976 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.829833 26691 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:56.831069 26899 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.831590 26691 ts_tablet_manager.cc:1367] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.124s user 0.030s sys 0.011s
I20240502 08:05:56.832255 26542 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.822314 26904 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.808506 26398 catalog_manager.cc:5484] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:56.840252 26780 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:56.834233 26691 raft_consensus.cc:348] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.853513 26691 raft_consensus.cc:374] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.853837 26691 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.854468 26691 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.838505 26976 raft_consensus.cc:3037] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.859681 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.861080 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.861917 26496 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.869038 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.869704 26495 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.869968 26892 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.870934 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.871716 26501 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.874883 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.876309 26541 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:56.856151 26780 raft_consensus.cc:686] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:56.881186 26904 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:56.887769 26887 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:56.888450 26920 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.888271 26887 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.866014 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.889358 26644 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:56.863658 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.890170 26887 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.890504 26498 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:56.893862 26892 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:56.894032 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.897192 26899 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.899243 26899 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:56.899961 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.902498 26691 ts_tablet_manager.cc:1398] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.071s user 0.000s sys 0.005s
I20240502 08:05:56.904014 26691 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
W20240502 08:05:56.905891 26467 leader_election.cc:341] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.906122 26571 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:56.902725 26716 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 08:05:56.911075 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.911701 26498 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.908190 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.912787 26570 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:56.914868 26501 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:56.915563 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:56.918546 26642 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.918632 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:56.919548 26496 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
W20240502 08:05:56.922472 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:56.926695 26780 consensus_queue.cc:237] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.927628 26693 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:56.928264 26693 ts_tablet_manager.cc:1367] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.240s user 0.058s sys 0.031s
I20240502 08:05:56.930744 26693 raft_consensus.cc:348] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.925122 26615 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.937371 26893 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.937781 26893 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.938195 26893 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.939399 26693 raft_consensus.cc:374] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:56.939705 26693 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:56.934367 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:56.940659 26693 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:56.941568 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.941429 26886 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 08:05:56.948884 26467 leader_election.cc:341] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.952886 26693 ts_tablet_manager.cc:1398] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20240502 08:05:56.953583 26693 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
W20240502 08:05:56.954857 26467 leader_election.cc:341] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:56.955179 26467 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:56.948341 26886 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.948065 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:56.950543 26889 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:56.963687 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:56.959971 26886 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.965783 26889 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:56.975477 26576 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:56.978436 26885 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:56.979585 26498 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:56.970290 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:56.982609 26690 tablet_bootstrap.cc:654] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:56.977032 26889 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:56.982432 26615 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:56.991478 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:56.992797 26398 catalog_manager.cc:5484] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:56.991264 26716 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 08:05:57.007697 26616 leader_election.cc:341] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.009220 26920 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:56.993422 26976 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.011263 26899 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.018591 26691 tablet_bootstrap.cc:654] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.025642 26885 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:57.027493 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:57.028095 26647 raft_consensus.cc:3037] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.028194 26889 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.030354 26501 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:57.034824 26889 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.020124 26716 raft_consensus.cc:686] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:57.024474 26976 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.046769 26893 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.048050 26645 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:57.050812 26893 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.051551 26893 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.051633 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.052479 26572 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:57.056798 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.058012 26501 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:57.060534 26615 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.061657 26893 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.062929 26693 tablet_bootstrap.cc:654] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.069135 26886 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.034744 26892 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.084569 26690 tablet_bootstrap.cc:492] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
W20240502 08:05:57.084507 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 63528.9us
I20240502 08:05:57.085369 26690 ts_tablet_manager.cc:1367] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.258s user 0.017s sys 0.007s
I20240502 08:05:57.089175 26886 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.089906 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.090517 26500 raft_consensus.cc:3037] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.082015 26892 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.038664 26716 consensus_queue.cc:237] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.096997 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.097791 26572 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:57.098320 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.098977 26570 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.098389 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:57.092442 26690 raft_consensus.cc:348] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.100258 26690 raft_consensus.cc:374] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.100654 26690 raft_consensus.cc:729] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.101189 26541 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:57.102588 26647 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:57.104965 26780 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.104928 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:57.106443 26642 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:57.108520 26690 consensus_queue.cc:260] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.111249 26542 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.112609 26976 raft_consensus.cc:2781] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.113068 26976 raft_consensus.cc:686] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:57.113865 26976 consensus_queue.cc:237] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.116603 26500 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:57.117828 26690 ts_tablet_manager.cc:1398] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.032s user 0.004s sys 0.002s
I20240502 08:05:57.118729 26690 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
W20240502 08:05:57.108409 26467 leader_election.cc:341] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.126216 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.123708 26892 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.129401 26886 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.131587 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.136548 26500 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.138299 26615 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.128907 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.139132 26892 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.139483 26892 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.139820 26892 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.142184 26690 tablet_bootstrap.cc:654] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.148111 26886 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20240502 08:05:57.156108 26616 leader_election.cc:341] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.167318 26691 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.167815 26691 ts_tablet_manager.cc:1367] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.264s user 0.013s sys 0.012s
I20240502 08:05:57.148936 26570 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:57.182374 26691 raft_consensus.cc:348] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.183107 26691 raft_consensus.cc:374] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.183447 26691 raft_consensus.cc:729] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.184078 26691 consensus_queue.cc:260] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.186488 26691 ts_tablet_manager.cc:1398] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20240502 08:05:57.187366 26691 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.190366 26693 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.190825 26693 ts_tablet_manager.cc:1367] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.237s user 0.015s sys 0.010s
I20240502 08:05:57.192323 26396 catalog_manager.cc:5484] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:57.210677 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:57.211500 26885 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.212046 26885 raft_consensus.cc:686] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:57.212894 26885 consensus_queue.cc:237] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.217792 26887 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.209794 26892 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.220214 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.229416 26501 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.218073 26395 catalog_manager.cc:5484] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:57.221418 26837 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240502 08:05:57.240638 26501 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:57.241793 26615 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.242830 26837 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.243568 26837 raft_consensus.cc:686] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:57.227677 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.245179 26837 consensus_queue.cc:237] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.218271 26887 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:57.246819 26616 leader_election.cc:341] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.248299 26887 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.249022 26691 tablet_bootstrap.cc:654] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.252627 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.227751 26892 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.224452 26693 raft_consensus.cc:348] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.255091 26693 raft_consensus.cc:374] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.255381 26693 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
W20240502 08:05:57.254215 26467 leader_election.cc:341] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.254405 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.258862 26896 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 08:05:57.262945 26467 leader_election.cc:341] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.263374 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.264147 26982 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.265215 26693 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.266110 26887 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.266501 26691 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.266558 26887 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.267130 26691 ts_tablet_manager.cc:1367] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.001s
I20240502 08:05:57.268358 26690 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.268911 26690 ts_tablet_manager.cc:1367] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.150s user 0.013s sys 0.007s
I20240502 08:05:57.269227 26887 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.269655 26691 raft_consensus.cc:348] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.270373 26691 raft_consensus.cc:374] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.270637 26691 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.271291 26691 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.271807 26690 raft_consensus.cc:348] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.272678 26690 raft_consensus.cc:374] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.272977 26690 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.273316 26691 ts_tablet_manager.cc:1398] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:57.274199 26690 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.276557 26691 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.280865 26396 catalog_manager.cc:5484] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:57.281934 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.282461 26690 ts_tablet_manager.cc:1398] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 08:05:57.283704 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:57.285920 26467 leader_election.cc:341] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:57.286576 26467 leader_election.cc:341] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.286904 26467 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.288559 26690 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.288913 26982 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.291158 26693 ts_tablet_manager.cc:1398] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.100s user 0.003s sys 0.003s
I20240502 08:05:57.292523 26395 catalog_manager.cc:5484] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:57.298256 26693 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:57.300794 26691 tablet_bootstrap.cc:654] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.304853 26690 tablet_bootstrap.cc:654] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.319972 26837 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.321991 26837 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.323948 26837 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.319079 26780 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.325652 26780 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.331966 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.332651 26498 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.334661 26615 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.335613 26837 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.336011 26837 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.336444 26837 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.336129 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.337162 26572 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.342487 26691 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.345120 26691 ts_tablet_manager.cc:1367] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.006s
I20240502 08:05:57.344048 26780 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.345806 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.347743 26691 raft_consensus.cc:348] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.348573 26691 raft_consensus.cc:374] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.349002 26691 raft_consensus.cc:729] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.344131 26837 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.353107 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.353183 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.354012 26498 raft_consensus.cc:2379] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:57.354074 26837 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.354305 26501 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.359851 26541 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:57.361339 26501 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:57.362756 26693 tablet_bootstrap.cc:654] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.362707 26615 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.363551 26837 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.354619 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.365178 26576 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.354481 26691 consensus_queue.cc:260] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.366289 26976 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.369258 26691 ts_tablet_manager.cc:1398] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20240502 08:05:57.372519 26837 raft_consensus.cc:686] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:57.373445 26691 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.372119 26576 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:57.373364 26837 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.382061 26691 tablet_bootstrap.cc:654] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.393134 26976 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.399784 26690 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.400182 26690 ts_tablet_manager.cc:1367] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.112s user 0.042s sys 0.047s
I20240502 08:05:57.401015 26396 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:57.402123 26690 raft_consensus.cc:348] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.402824 26690 raft_consensus.cc:374] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.403079 26690 raft_consensus.cc:729] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.403481 26693 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.403987 26693 ts_tablet_manager.cc:1367] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.003s
I20240502 08:05:57.403656 26690 consensus_queue.cc:260] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.405649 26690 ts_tablet_manager.cc:1398] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240502 08:05:57.406334 26693 raft_consensus.cc:348] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.406849 26690 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.407017 26693 raft_consensus.cc:374] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.407285 26693 raft_consensus.cc:729] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.407939 26693 consensus_queue.cc:260] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.410269 26693 ts_tablet_manager.cc:1398] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:57.412055 26693 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:57.416554 26690 tablet_bootstrap.cc:654] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.421893 26691 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.422410 26691 ts_tablet_manager.cc:1367] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.004s
I20240502 08:05:57.424595 26691 raft_consensus.cc:348] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.425271 26691 raft_consensus.cc:374] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.424962 26976 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:57.426278 26691 raft_consensus.cc:729] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.426955 26691 consensus_queue.cc:260] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.428503 26693 tablet_bootstrap.cc:654] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.428880 26691 ts_tablet_manager.cc:1398] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 08:05:57.439582 26691 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.439790 26693 tablet_bootstrap.cc:492] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.440240 26693 ts_tablet_manager.cc:1367] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.002s
I20240502 08:05:57.441968 26690 tablet_bootstrap.cc:492] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.442416 26690 ts_tablet_manager.cc:1367] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20240502 08:05:57.442997 26693 raft_consensus.cc:348] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.443941 26693 raft_consensus.cc:374] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.444242 26693 raft_consensus.cc:729] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.444766 26691 tablet_bootstrap.cc:654] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.444878 26693 consensus_queue.cc:260] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.448230 26693 ts_tablet_manager.cc:1398] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:05:57.449119 26693 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:57.459551 26690 raft_consensus.cc:348] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.460886 26690 raft_consensus.cc:374] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.461237 26690 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.479156 26690 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.480639 26976 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.483929 26690 ts_tablet_manager.cc:1398] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20240502 08:05:57.484791 26690 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.486227 26693 tablet_bootstrap.cc:654] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.490504 26690 tablet_bootstrap.cc:654] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.495364 26691 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.495807 26691 ts_tablet_manager.cc:1367] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.013s
I20240502 08:05:57.497835 26691 raft_consensus.cc:348] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.498521 26691 raft_consensus.cc:374] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.498781 26691 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.503005 26690 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.503461 26690 ts_tablet_manager.cc:1367] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20240502 08:05:57.505982 26690 raft_consensus.cc:348] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.507014 26690 raft_consensus.cc:374] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.507337 26690 raft_consensus.cc:729] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.508209 26690 consensus_queue.cc:260] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.510152 26690 ts_tablet_manager.cc:1398] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 08:05:57.510974 26690 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.512285 26693 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.512713 26693 ts_tablet_manager.cc:1367] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.008s
I20240502 08:05:57.515081 26693 raft_consensus.cc:348] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.515735 26693 raft_consensus.cc:374] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.516045 26693 raft_consensus.cc:729] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.515519 26691 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.516198 26690 tablet_bootstrap.cc:654] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.516693 26693 consensus_queue.cc:260] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.517570 26691 ts_tablet_manager.cc:1398] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20240502 08:05:57.518440 26691 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.519122 26693 ts_tablet_manager.cc:1398] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240502 08:05:57.520074 26693 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:57.530827 26693 tablet_bootstrap.cc:654] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.533372 26982 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.533831 26982 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.534682 26691 tablet_bootstrap.cc:654] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.536401 26982 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.538818 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.539216 26690 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.539636 26690 ts_tablet_manager.cc:1367] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.005s
I20240502 08:05:57.540019 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.541167 26647 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:57.542272 26467 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:05:57.543040 26467 leader_election.cc:341] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.543778 26889 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.544306 26889 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.543778 26690 raft_consensus.cc:348] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.544663 26889 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.544893 26690 raft_consensus.cc:374] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.545181 26690 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.545774 26690 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.548235 26690 ts_tablet_manager.cc:1398] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240502 08:05:57.548439 26691 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.548913 26691 ts_tablet_manager.cc:1367] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.002s
I20240502 08:05:57.549839 26690 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.551182 26691 raft_consensus.cc:348] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.551239 26889 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.551898 26691 raft_consensus.cc:374] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.552213 26691 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.552971 26691 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.553920 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.554973 26691 ts_tablet_manager.cc:1398] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20240502 08:05:57.555150 26467 leader_election.cc:341] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.555356 26690 tablet_bootstrap.cc:654] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.561533 26889 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.563664 26691 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.566934 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:57.567756 26647 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.572283 26837 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.572561 26691 tablet_bootstrap.cc:654] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.572844 26837 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.575230 26837 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.576776 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.578796 26498 raft_consensus.cc:2454] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.580072 26615 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.577292 26647 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:57.578667 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.581171 26693 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.581538 26467 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.581571 26576 raft_consensus.cc:2454] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.582396 26887 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.583108 26887 raft_consensus.cc:686] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:57.582015 26837 raft_consensus.cc:2781] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.584035 26837 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.584499 26837 raft_consensus.cc:3037] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.586084 26982 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.586534 26982 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.588423 26982 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.589497 26896 consensus_queue.cc:1035] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.590451 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:57.591953 26467 leader_election.cc:341] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.593485 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.584102 26887 consensus_queue.cc:237] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:57.594743 26467 leader_election.cc:341] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.595166 26467 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.595831 26885 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.581820 26693 ts_tablet_manager.cc:1367] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.002s
I20240502 08:05:57.607197 26887 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.609225 26887 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.611464 26837 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.613051 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.613564 26837 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.613866 26576 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:57.615136 26467 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.616191 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.616814 26572 raft_consensus.cc:3037] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.617455 26889 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:05:57.617974 26889 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.618420 26889 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.624011 26572 raft_consensus.cc:2454] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:57.625093 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.625717 26500 raft_consensus.cc:3037] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.627401 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.628264 26647 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:57.628723 26616 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.629545 26894 raft_consensus.cc:2781] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:57.629992 26894 raft_consensus.cc:686] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:57.631155 26894 consensus_queue.cc:237] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.632444 26500 raft_consensus.cc:2454] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
W20240502 08:05:57.646975 26467 consensus_peers.cc:485] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa -> Peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Couldn't send request to peer 755f35bc66a841f382d1c4df6c6f6b64. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:57.629945 26887 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.649165 26920 consensus_queue.cc:1035] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.650947 26889 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.653020 26691 tablet_bootstrap.cc:492] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.653519 26691 ts_tablet_manager.cc:1367] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.001s
I20240502 08:05:57.653556 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.654251 26576 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.655009 26978 consensus_queue.cc:1035] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.661245 26693 raft_consensus.cc:348] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.662009 26693 raft_consensus.cc:374] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.662285 26693 raft_consensus.cc:729] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.662817 26693 consensus_queue.cc:260] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.665450 26690 tablet_bootstrap.cc:492] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.665932 26690 ts_tablet_manager.cc:1367] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.116s user 0.026s sys 0.059s
I20240502 08:05:57.665915 26576 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:05:57.668494 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:57.669109 26643 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.669325 26467 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.670145 26887 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:57.670765 26887 raft_consensus.cc:686] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:57.670636 26690 raft_consensus.cc:348] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.671353 26690 raft_consensus.cc:374] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.671631 26690 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.671666 26887 consensus_queue.cc:237] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.672327 26690 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.667039 26691 raft_consensus.cc:348] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.674190 26691 raft_consensus.cc:374] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.674451 26691 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.675037 26691 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.668088 26889 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.676048 26398 catalog_manager.cc:5484] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:57.678175 26690 ts_tablet_manager.cc:1398] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:05:57.679070 26690 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.680083 26693 ts_tablet_manager.cc:1398] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20240502 08:05:57.680894 26693 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:57.681557 26978 consensus_queue.cc:1035] T d0b9fcbac5f24ae8b0c8f071ae34229f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.677116 26691 ts_tablet_manager.cc:1398] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20240502 08:05:57.683584 26691 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.709266 26576 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.718444 26690 tablet_bootstrap.cc:654] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.681499 26395 catalog_manager.cc:5484] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:57.720851 26643 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:05:57.734123 26889 consensus_queue.cc:1035] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:57.739734 26920 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.744732 26691 tablet_bootstrap.cc:654] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.745090 26693 tablet_bootstrap.cc:654] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.746726 26837 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.747625 26837 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.761770 26837 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.762971 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.776665 26716 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.777364 26716 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.779935 26398 catalog_manager.cc:5484] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 2, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:57.764081 26501 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.764670 26887 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.789590 26690 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:57.790077 26690 ts_tablet_manager.cc:1367] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.111s user 0.020s sys 0.009s
I20240502 08:05:57.793336 26690 raft_consensus.cc:348] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.794015 26690 raft_consensus.cc:374] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.794301 26690 raft_consensus.cc:729] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.795032 26690 consensus_queue.cc:260] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.772473 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.824585 26716 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.825517 26690 ts_tablet_manager.cc:1398] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.035s user 0.001s sys 0.004s
I20240502 08:05:57.826563 26690 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:57.809163 26615 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.827195 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.809547 26576 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.837371 26691 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:57.798385 26887 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.838191 26920 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.842581 26920 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.843998 26887 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.844367 26920 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.844884 26920 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.841533 26691 ts_tablet_manager.cc:1367] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.158s user 0.014s sys 0.008s
I20240502 08:05:57.907366 26693 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:57.846071 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.921057 26693 ts_tablet_manager.cc:1367] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.240s user 0.020s sys 0.011s
I20240502 08:05:57.897789 26780 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.922348 26615 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:57.922206 26780 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.923154 26716 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.870311 26904 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.934064 26691 raft_consensus.cc:348] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.934388 26904 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.934835 26691 raft_consensus.cc:374] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.935144 26691 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.936069 26904 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.937031 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.853461 26894 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.070s
W20240502 08:05:57.938565 26542 leader_election.cc:341] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.935765 26691 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.847018 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.939589 26496 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.940004 26691 ts_tablet_manager.cc:1398] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240502 08:05:57.940871 26691 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:57.942066 26615 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:57.942313 26780 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.845221 26920 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.942829 26894 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.942963 26920 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.943218 26894 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.896696 26976 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:57.943354 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.943614 26894 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.943881 26976 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.944751 26896 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:57.946537 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.951277 26976 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:57.952961 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.953619 26645 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:57.954790 26542 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.956254 26976 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.956007 26920 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.956619 26976 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.957022 26976 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.958494 26920 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.958804 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.959414 26500 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.959298 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.959962 26496 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:57.872902 26892 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20240502 08:05:57.846426 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.925004 26693 raft_consensus.cc:348] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.964058 26693 raft_consensus.cc:374] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:57.964107 26576 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:57.964437 26693 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:57.965112 26693 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.966502 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.867815 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:57.851536 26690 tablet_bootstrap.cc:654] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.894097 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:57.968389 26572 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:57.969161 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:57.969780 26576 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.971055 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:57.971711 26501 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:57.972672 26887 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:57.973033 26887 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:57.973366 26887 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:57.976194 26576 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:57.976053 26894 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:57.977296 26642 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:57.982589 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:57.984787 26691 tablet_bootstrap.cc:654] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:57.987710 26693 ts_tablet_manager.cc:1398] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
I20240502 08:05:57.988703 26693 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
W20240502 08:05:57.989567 26541 leader_election.cc:341] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:57.989964 26541 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:57.992291 26894 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:57.994068 26541 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:57.997561 26616 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:57.993464 26780 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:57.998994 26904 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.004979 26904 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.005321 26904 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.012172 26500 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:58.012115 26887 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.015775 26501 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:58.016781 26887 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:58.017341 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:58.021888 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.022730 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:58.023653 26643 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:58.006865 26892 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.033722 26693 tablet_bootstrap.cc:654] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.033692 26976 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.037876 26889 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240502 08:05:58.040588 26904 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.042968 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.051683 26570 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:58.051671 26497 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.055425 26690 tablet_bootstrap.cc:492] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.056528 26690 ts_tablet_manager.cc:1367] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.230s user 0.028s sys 0.035s
I20240502 08:05:58.059002 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.059616 26647 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.062166 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.038364 26892 raft_consensus.cc:686] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:58.039600 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:58.048429 26976 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.064800 26982 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.058109 26889 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.071792 26691 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.072270 26691 ts_tablet_manager.cc:1367] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.004s
I20240502 08:05:58.074671 26691 raft_consensus.cc:348] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.075310 26691 raft_consensus.cc:374] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.075592 26691 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.076599 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.077399 26498 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:58.079231 26690 raft_consensus.cc:348] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.078155 26691 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.080806 26904 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.071688 26889 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.088071 26889 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.088375 26691 ts_tablet_manager.cc:1398] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 08:05:58.089298 26691 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.081558 26894 consensus_queue.cc:1035] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:05:58.084458 26693 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:58.095275 26693 ts_tablet_manager.cc:1367] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.107s user 0.014s sys 0.003s
I20240502 08:05:58.095888 26647 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.090342 26892 consensus_queue.cc:237] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.097673 26693 raft_consensus.cc:348] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.098392 26693 raft_consensus.cc:374] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.084424 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.084962 26690 raft_consensus.cc:374] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.098660 26693 raft_consensus.cc:729] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.098865 26690 raft_consensus.cc:729] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.100136 26690 consensus_queue.cc:260] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.101258 26693 consensus_queue.cc:260] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.100009 26575 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 1: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:58.103991 26642 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:58.105664 26693 ts_tablet_manager.cc:1398] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20240502 08:05:58.108340 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.101009 26497 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.115589 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.116461 26893 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.114260 26716 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.103183 26542 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:05:58.119266 26467 leader_election.cc:341] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.119370 26904 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.119882 26904 raft_consensus.cc:686] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:58.120702 26904 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.121237 26542 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:58.122006 26780 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.118909 26716 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.127092 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.129881 26690 ts_tablet_manager.cc:1398] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
W20240502 08:05:58.130925 26467 leader_election.cc:341] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.128779 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.131832 26467 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.132089 26693 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:58.144762 26691 tablet_bootstrap.cc:654] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.136766 26896 consensus_queue.cc:1035] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 08:05:58.174516 26616 leader_election.cc:341] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.168902 26889 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.177845 26690 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.157063 26716 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.213960 26572 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.214172 26576 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:05:58.217907 26398 catalog_manager.cc:5484] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:58.198442 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.231559 26691 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.232579 26691 ts_tablet_manager.cc:1367] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.143s user 0.002s sys 0.015s
I20240502 08:05:58.236325 26643 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:58.237331 26615 leader_election.cc:341] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.237756 26615 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.239365 26892 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.232637 26885 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 08:05:58.292608 26920 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.301256 26693 tablet_bootstrap.cc:654] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.304107 26690 tablet_bootstrap.cc:654] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.293576 26920 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.311539 26920 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.314317 26395 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:58.319200 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.322510 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:58.324532 26616 leader_election.cc:341] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:58.327016 26615 leader_election.cc:341] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.327931 26615 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.331115 26837 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.334681 26691 raft_consensus.cc:348] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.335443 26691 raft_consensus.cc:374] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.335741 26691 raft_consensus.cc:729] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.336364 26691 consensus_queue.cc:260] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.341343 26647 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:05:58.352959 26885 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:58.369953 26982 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:58.370030 26691 ts_tablet_manager.cc:1398] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.137s user 0.001s sys 0.005s
I20240502 08:05:58.383435 26691 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.389528 26690 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.395327 26690 ts_tablet_manager.cc:1367] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.222s user 0.021s sys 0.005s
I20240502 08:05:58.397668 26690 raft_consensus.cc:348] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.398216 26690 raft_consensus.cc:374] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.398420 26690 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.398936 26690 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.401093 26690 ts_tablet_manager.cc:1398] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:05:58.401961 26690 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.409413 26690 tablet_bootstrap.cc:654] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.418383 26690 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.418895 26690 ts_tablet_manager.cc:1367] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240502 08:05:58.419747 26691 tablet_bootstrap.cc:654] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.420608 26690 raft_consensus.cc:348] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.421095 26690 raft_consensus.cc:374] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.421288 26690 raft_consensus.cc:729] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.421742 26690 consensus_queue.cc:260] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.423705 26690 ts_tablet_manager.cc:1398] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:05:58.425032 26690 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.430259 26690 tablet_bootstrap.cc:654] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.435230 26691 tablet_bootstrap.cc:492] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.435716 26691 ts_tablet_manager.cc:1367] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.003s
I20240502 08:05:58.437961 26691 raft_consensus.cc:348] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.438691 26691 raft_consensus.cc:374] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.439005 26691 raft_consensus.cc:729] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.439671 26691 consensus_queue.cc:260] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.441759 26691 ts_tablet_manager.cc:1398] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:58.442721 26691 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.448901 26691 tablet_bootstrap.cc:654] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.452948 26693 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:58.454869 26693 ts_tablet_manager.cc:1367] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.323s user 0.031s sys 0.017s
I20240502 08:05:58.460633 26691 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.461205 26691 ts_tablet_manager.cc:1367] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240502 08:05:58.463586 26691 raft_consensus.cc:348] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.464442 26691 raft_consensus.cc:374] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.464746 26691 raft_consensus.cc:729] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.465549 26691 consensus_queue.cc:260] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.467916 26691 ts_tablet_manager.cc:1398] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:58.469141 26693 raft_consensus.cc:348] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.469641 26693 raft_consensus.cc:374] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.469900 26693 raft_consensus.cc:729] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.470536 26693 consensus_queue.cc:260] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.472493 26693 ts_tablet_manager.cc:1398] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 08:05:58.473392 26691 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.473986 26837 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.474637 26837 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.477162 26837 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.480291 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:58.482215 26616 leader_election.cc:341] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.484616 26691 tablet_bootstrap.cc:654] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.486716 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:58.490139 26615 leader_election.cc:341] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.490525 26615 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.491222 26693 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:58.491693 26920 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.492247 26920 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.492623 26920 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.496349 26920 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.497723 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.498387 26500 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:58.499691 26615 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:58.501802 26716 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.502218 26716 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.502547 26716 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.504685 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:05:58.506531 26616 leader_election.cc:341] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.516984 26690 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.517501 26690 ts_tablet_manager.cc:1367] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.093s user 0.023s sys 0.032s
I20240502 08:05:58.519989 26690 raft_consensus.cc:348] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.520519 26690 raft_consensus.cc:374] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.521050 26690 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.521678 26690 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.522578 26885 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.523026 26885 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.523912 26690 ts_tablet_manager.cc:1398] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:05:58.524380 26693 tablet_bootstrap.cc:654] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.524734 26690 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.526991 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.528695 26976 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.529172 26976 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.530833 26690 tablet_bootstrap.cc:654] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
W20240502 08:05:58.531389 26467 leader_election.cc:341] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.520501 26716 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.535010 26716 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.537614 26976 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.539721 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.540365 26643 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:58.542394 26542 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.542929 26691 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.543512 26691 ts_tablet_manager.cc:1367] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.070s user 0.040s sys 0.022s
I20240502 08:05:58.544729 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:05:58.546290 26616 leader_election.cc:341] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.546764 26691 raft_consensus.cc:348] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.547683 26691 raft_consensus.cc:374] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.548007 26691 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.548714 26691 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.551290 26691 ts_tablet_manager.cc:1398] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:05:58.552299 26885 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.553157 26976 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.553674 26976 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.554199 26976 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.556499 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.559669 26837 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.560117 26837 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.561828 26837 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.562515 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.563110 26496 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:58.563323 26467 leader_election.cc:341] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.564042 26467 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.570457 26496 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:58.566169 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.577584 26887 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.579203 26887 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.581039 26887 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.581496 26887 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.578083 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.581637 26893 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:58.582370 26571 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:58.584776 26615 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.588652 26693 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:58.588503 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.589097 26693 ts_tablet_manager.cc:1367] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.098s user 0.034s sys 0.003s
I20240502 08:05:58.589351 26500 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:58.591229 26616 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.592123 26893 raft_consensus.cc:2781] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.592468 26893 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.592792 26893 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.594995 26920 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.596962 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.599100 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.600790 26920 raft_consensus.cc:686] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
W20240502 08:05:58.591375 26541 leader_election.cc:341] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:58.604215 26467 leader_election.cc:341] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:58.608134 26467 leader_election.cc:341] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.608633 26467 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.605448 26920 consensus_queue.cc:237] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.552356 26691 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.613350 26893 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.626595 26887 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:58.628243 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:58.621337 26693 raft_consensus.cc:348] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.629194 26693 raft_consensus.cc:374] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.629478 26693 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.630007 26976 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.631170 26693 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.632037 26570 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.636869 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.638558 26691 tablet_bootstrap.cc:654] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.638726 26716 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240502 08:05:58.637790 26496 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.665496 26976 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.628299 26893 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:58.645437 26570 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:58.668686 26396 catalog_manager.cc:5484] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:58.673846 26693 ts_tablet_manager.cc:1398] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.084s user 0.005s sys 0.001s
I20240502 08:05:58.675304 26693 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:58.639766 26904 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.680474 26904 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.683234 26904 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.692615 26496 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:58.694219 26690 tablet_bootstrap.cc:492] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.694677 26690 ts_tablet_manager.cc:1367] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.170s user 0.073s sys 0.078s
I20240502 08:05:58.694705 26615 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:58.694989 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.695881 26893 raft_consensus.cc:2781] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.696261 26893 raft_consensus.cc:686] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:58.697016 26893 consensus_queue.cc:237] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.700052 26690 raft_consensus.cc:348] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.703374 26690 raft_consensus.cc:374] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.703639 26690 raft_consensus.cc:729] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.704296 26690 consensus_queue.cc:260] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.706413 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.706454 26690 ts_tablet_manager.cc:1398] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240502 08:05:58.707816 26690 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.707968 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.708546 26643 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.708946 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.709584 26642 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:58.712749 26690 tablet_bootstrap.cc:654] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.716637 26643 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.717916 26690 tablet_bootstrap.cc:492] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.718374 26690 ts_tablet_manager.cc:1367] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240502 08:05:58.720252 26542 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.721143 27062 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.721493 27062 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.721674 26542 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.721856 27062 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.722657 26904 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.723099 26904 raft_consensus.cc:686] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:58.723896 26904 consensus_queue.cc:237] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.728256 26691 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.728669 26691 ts_tablet_manager.cc:1367] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.176s user 0.008s sys 0.003s
I20240502 08:05:58.721741 27063 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240502 08:05:58.733816 26976 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:58.737092 26691 raft_consensus.cc:348] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.738629 26691 raft_consensus.cc:374] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.738844 26691 raft_consensus.cc:729] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.739399 26691 consensus_queue.cc:260] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.758936 27063 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.759617 27063 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.769354 26691 ts_tablet_manager.cc:1398] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.040s user 0.004s sys 0.003s
I20240502 08:05:58.771044 27063 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.772910 27062 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.775828 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.776475 26647 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.778375 26693 tablet_bootstrap.cc:654] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.771747 26691 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.772472 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.780336 26396 catalog_manager.cc:5484] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:58.770270 26690 raft_consensus.cc:348] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.786756 26690 raft_consensus.cc:374] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.788081 26399 catalog_manager.cc:5484] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:58.787068 26690 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.789588 26690 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.791517 26690 ts_tablet_manager.cc:1398] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.072s user 0.013s sys 0.005s
I20240502 08:05:58.792716 27062 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.793498 26647 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.794762 26542 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.796113 27062 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.793509 26691 tablet_bootstrap.cc:654] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.797380 27062 raft_consensus.cc:686] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:58.798375 27062 consensus_queue.cc:237] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:05:58.806012 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 85505.7us
W20240502 08:05:58.806705 26541 leader_election.cc:341] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:58.807489 26541 leader_election.cc:341] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:58.807588 26693 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:58.808041 26693 ts_tablet_manager.cc:1367] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.001s
I20240502 08:05:58.810281 26690 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.816735 26693 raft_consensus.cc:348] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.817462 26693 raft_consensus.cc:374] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.817726 26693 raft_consensus.cc:729] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.818415 26693 consensus_queue.cc:260] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.821341 26693 ts_tablet_manager.cc:1398] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20240502 08:05:58.823964 26690 tablet_bootstrap.cc:654] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.824142 26691 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:58.824549 26691 ts_tablet_manager.cc:1367] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.007s
I20240502 08:05:58.827389 26693 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:58.826956 26691 raft_consensus.cc:348] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.827694 26691 raft_consensus.cc:374] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.827951 26691 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.828773 26691 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.831377 26691 ts_tablet_manager.cc:1398] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20240502 08:05:58.832437 26691 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:58.837731 27068 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.837881 27062 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.839241 27068 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.839650 27062 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.841104 27068 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.841425 27062 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.838330 26691 tablet_bootstrap.cc:654] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.842856 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.843593 26643 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:58.844435 26396 catalog_manager.cc:5484] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:58.846968 26542 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.848766 26690 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.849189 26690 ts_tablet_manager.cc:1367] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.000s
I20240502 08:05:58.850306 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.850899 26647 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.851516 26690 raft_consensus.cc:348] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.851866 26542 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.852216 26690 raft_consensus.cc:374] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:58.852499 26690 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:58.852669 27068 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:58.853011 27068 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.853364 27068 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.853201 26690 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.854213 26976 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:05:58.854554 26976 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:58.854907 26976 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:58.855284 26690 ts_tablet_manager.cc:1398] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20240502 08:05:58.856153 26690 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:58.861119 26690 tablet_bootstrap.cc:654] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.860738 26976 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.862763 26976 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.833879 26693 tablet_bootstrap.cc:654] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:58.864280 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.865082 26643 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:58.876853 27068 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.892300 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:58.893182 26647 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:58.897661 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.898820 26496 raft_consensus.cc:2379] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:58.897732 27068 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.880270 26643 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:05:58.901690 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:58.901717 26542 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.903110 27068 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:58.903708 27068 raft_consensus.cc:686] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:58.904733 27068 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.922255 26885 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:58.922899 26885 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.925232 26885 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:58.947793 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.948700 26497 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.953060 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:58.954191 26643 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:05:58.970954 26693 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:58.975965 26693 ts_tablet_manager.cc:1367] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.149s user 0.014s sys 0.001s
I20240502 08:05:58.971007 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:58.977676 26647 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:58.979665 26542 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:58.980569 26976 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:58.981053 26976 raft_consensus.cc:686] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:05:58.981920 26976 consensus_queue.cc:237] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:58.983321 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:58.985141 26982 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:58.992012 26690 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:58.992527 26690 ts_tablet_manager.cc:1367] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.137s user 0.028s sys 0.060s
I20240502 08:05:58.994570 26982 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:58.997501 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:58.998327 26497 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:58.999215 26691 tablet_bootstrap.cc:492] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:59.000478 26691 ts_tablet_manager.cc:1367] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.168s user 0.051s sys 0.066s
I20240502 08:05:59.002190 26690 raft_consensus.cc:348] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.003018 26690 raft_consensus.cc:374] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.003309 26690 raft_consensus.cc:729] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.003930 26690 consensus_queue.cc:260] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.005555 26399 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 2, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:59.008021 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.008333 26691 raft_consensus.cc:348] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.009171 26691 raft_consensus.cc:374] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.009482 26691 raft_consensus.cc:729] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.000361 26982 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.010249 26691 consensus_queue.cc:260] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.013294 26541 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.016633 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.017236 26497 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.018368 26976 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.021452 26496 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:05:59.030714 26976 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:59.032687 26690 ts_tablet_manager.cc:1398] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.040s user 0.008s sys 0.000s
I20240502 08:05:59.033411 26690 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:59.031674 26497 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:05:59.010090 26982 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.039724 26982 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:59.040470 26541 leader_election.cc:341] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.041009 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.012169 26693 raft_consensus.cc:348] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.043239 26693 raft_consensus.cc:374] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.043532 26693 raft_consensus.cc:729] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.032429 26691 ts_tablet_manager.cc:1398] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.031s user 0.003s sys 0.002s
I20240502 08:05:59.044070 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.044809 26691 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
W20240502 08:05:59.045461 26467 leader_election.cc:341] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.044698 26647 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:59.046773 26467 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:59.047689 26982 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.047343 26693 consensus_queue.cc:260] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.048135 26982 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.048573 26982 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.049903 26693 ts_tablet_manager.cc:1398] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.074s user 0.005s sys 0.001s
I20240502 08:05:59.037972 26690 tablet_bootstrap.cc:654] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.053118 26691 tablet_bootstrap.cc:654] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.076428 26693 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:59.079340 26982 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.082135 26693 tablet_bootstrap.cc:654] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.082851 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:05:59.088804 26467 leader_election.cc:341] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.092708 26982 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.094069 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:59.094626 26647 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.101756 26647 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:59.103152 26467 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:59.100029 26395 catalog_manager.cc:5484] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:59.103976 26885 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.106765 26885 raft_consensus.cc:686] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:59.106251 26690 tablet_bootstrap.cc:492] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:59.116767 26690 ts_tablet_manager.cc:1367] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.003s
W20240502 08:05:59.116413 26447 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:05:59.119495 26690 raft_consensus.cc:348] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.121013 26690 raft_consensus.cc:374] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.121345 26690 raft_consensus.cc:729] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.122221 26690 consensus_queue.cc:260] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.124153 26690 ts_tablet_manager.cc:1398] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:05:59.116344 26885 consensus_queue.cc:237] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.125317 26690 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:59.131495 26690 tablet_bootstrap.cc:654] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.139143 26716 consensus_queue.cc:1035] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.157572 27062 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.158448 27062 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.158701 26716 consensus_queue.cc:1035] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.160457 27062 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.161442 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.163816 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:59.164378 26541 leader_election.cc:341] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:05:59.165256 26542 leader_election.cc:341] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.165688 26542 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.166514 27068 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:59.167378 26690 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:59.167872 26690 ts_tablet_manager.cc:1367] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.017s
I20240502 08:05:59.169905 26690 raft_consensus.cc:348] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.170431 26690 raft_consensus.cc:374] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.170619 26690 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.171181 26690 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.172518 26690 ts_tablet_manager.cc:1398] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:05:59.173463 26690 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:59.179802 26690 tablet_bootstrap.cc:654] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.182896 26892 consensus_queue.cc:1035] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.190316 26690 tablet_bootstrap.cc:492] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:59.190764 26690 ts_tablet_manager.cc:1367] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240502 08:05:59.193187 26690 raft_consensus.cc:348] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.193912 26690 raft_consensus.cc:374] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.194224 26690 raft_consensus.cc:729] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.194792 26690 consensus_queue.cc:260] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.196319 26690 ts_tablet_manager.cc:1398] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:05:59.200537 26690 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Bootstrap starting.
I20240502 08:05:59.201752 27079 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.202370 27079 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.204492 27079 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.205715 26690 tablet_bootstrap.cc:654] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.214394 26691 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:59.219259 26691 ts_tablet_manager.cc:1367] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.175s user 0.023s sys 0.039s
I20240502 08:05:59.223515 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.224390 26568 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.215649 26693 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:59.227203 26693 ts_tablet_manager.cc:1367] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.151s user 0.026s sys 0.025s
W20240502 08:05:59.228639 26541 consensus_peers.cc:485] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f -> Peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Couldn't send request to peer 8f9fe1f3763547388c4301b13a127aaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:59.222519 26691 raft_consensus.cc:348] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.229700 26691 raft_consensus.cc:374] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.230172 26691 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.212302 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.233053 26497 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.234535 26616 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:59.240221 26892 raft_consensus.cc:2781] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.240569 26892 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.240906 26892 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.240763 26693 raft_consensus.cc:348] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.242240 26693 raft_consensus.cc:374] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.242563 26693 raft_consensus.cc:729] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.244275 26693 consensus_queue.cc:260] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.249981 26892 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.237965 26395 catalog_manager.cc:5484] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:59.234186 26691 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.251787 26691 ts_tablet_manager.cc:1398] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20240502 08:05:59.252593 26691 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:59.252810 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.252944 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.253353 26576 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.253698 26497 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.248198 26716 consensus_queue.cc:1035] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.250523 26693 ts_tablet_manager.cc:1398] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20240502 08:05:59.255160 26892 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.258206 26693 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:59.272667 26690 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: No bootstrap required, opened a new log
I20240502 08:05:59.273231 26690 ts_tablet_manager.cc:1367] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent bootstrapping tablet: real 0.073s user 0.025s sys 0.020s
I20240502 08:05:59.274542 26576 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.277878 26691 tablet_bootstrap.cc:654] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.277787 26616 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:59.278669 26690 raft_consensus.cc:348] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.279409 26690 raft_consensus.cc:374] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.279641 26716 raft_consensus.cc:2781] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.279747 26690 raft_consensus.cc:729] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.282189 26716 raft_consensus.cc:686] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:59.282559 26690 consensus_queue.cc:260] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.282958 26716 consensus_queue.cc:237] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.286396 26690 ts_tablet_manager.cc:1398] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240502 08:05:59.291210 26892 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.291677 26892 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.294214 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:05:59.295643 26615 leader_election.cc:341] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.295919 26693 tablet_bootstrap.cc:654] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.298243 26976 consensus_queue.cc:1035] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.299612 26497 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.306555 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.307215 26570 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.311460 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.312247 26920 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.312582 26920 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.312903 26920 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.325779 26920 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.340860 27063 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.361398 27063 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.366276 26920 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.326063 27080 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.368573 27080 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.353760 27079 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.370381 27079 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.373562 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.374461 26498 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.375901 26615 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.377171 27079 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.351771 26892 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.363602 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.365367 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.379261 26571 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:59.386902 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 49658.7us
I20240502 08:05:59.389689 26571 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.395496 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.396086 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.399272 26572 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 0.
I20240502 08:05:59.381811 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.402623 26570 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
W20240502 08:05:59.405174 26615 leader_election.cc:341] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.405869 26893 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.406252 26893 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.406543 26893 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.403419 26691 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:59.411990 26691 ts_tablet_manager.cc:1367] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.160s user 0.050s sys 0.033s
I20240502 08:05:59.415426 26691 raft_consensus.cc:348] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.416000 26693 tablet_bootstrap.cc:492] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:59.416857 26693 ts_tablet_manager.cc:1367] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.159s user 0.051s sys 0.049s
I20240502 08:05:59.416093 26691 raft_consensus.cc:374] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.417656 26893 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.418515 26691 raft_consensus.cc:729] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.419237 26691 consensus_queue.cc:260] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.420501 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.421170 26893 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.421237 26498 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.421998 26693 raft_consensus.cc:348] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.425382 26693 raft_consensus.cc:374] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.425644 26693 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.426329 26693 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.427593 26498 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.428194 26976 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.428483 26693 ts_tablet_manager.cc:1398] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20240502 08:05:59.422609 26691 ts_tablet_manager.cc:1398] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:05:59.423048 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.429634 26691 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64: Bootstrap starting.
I20240502 08:05:59.429771 26716 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.431126 26716 raft_consensus.cc:686] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:59.431840 26716 consensus_queue.cc:237] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.424741 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.428524 26615 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.433007 26572 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.434502 27079 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.434999 27079 raft_consensus.cc:686] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:59.435786 26691 tablet_bootstrap.cc:654] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.435775 27079 consensus_queue.cc:237] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.441115 26572 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.443588 26643 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:05:59.447194 26496 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.454931 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.455591 26498 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 0.
I20240502 08:05:59.457779 26497 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:05:59.462647 27062 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.467509 26693 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:59.469870 27063 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.472321 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:05:59.474256 26542 leader_election.cc:341] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.486703 26693 tablet_bootstrap.cc:654] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.477576 26395 catalog_manager.cc:5484] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:59.506350 26693 tablet_bootstrap.cc:492] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:59.506974 26693 ts_tablet_manager.cc:1367] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.003s
I20240502 08:05:59.514001 26693 raft_consensus.cc:348] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.514983 26693 raft_consensus.cc:374] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.515316 26693 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.516148 26693 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.521847 26693 ts_tablet_manager.cc:1398] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.014s user 0.003s sys 0.005s
I20240502 08:05:59.523036 26693 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:59.530200 26693 tablet_bootstrap.cc:654] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.533303 27079 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.533951 27079 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.536566 27079 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.539326 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.540127 26497 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.542021 26693 tablet_bootstrap.cc:492] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:59.542486 26693 ts_tablet_manager.cc:1367] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20240502 08:05:59.544920 26615 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.544839 26693 raft_consensus.cc:348] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.545553 26693 raft_consensus.cc:374] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.545809 26693 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.546135 26837 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.546464 26837 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.546453 26693 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.546919 26837 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.548300 26693 ts_tablet_manager.cc:1398] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:05:59.549149 26693 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Bootstrap starting.
I20240502 08:05:59.550248 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.551149 26570 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.554195 26693 tablet_bootstrap.cc:654] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:59.556466 26837 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.558547 26467 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:59.559931 26982 raft_consensus.cc:2781] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.560470 26982 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.560897 26982 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.559873 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.562794 26498 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.566118 26693 tablet_bootstrap.cc:492] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: No bootstrap required, opened a new log
I20240502 08:05:59.566583 26693 ts_tablet_manager.cc:1367] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240502 08:05:59.568045 27062 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.569298 26693 raft_consensus.cc:348] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.570014 26693 raft_consensus.cc:374] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.570300 26693 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.570847 26693 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.571452 26982 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.572010 26837 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.572713 26693 ts_tablet_manager.cc:1398] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 08:05:59.573335 26982 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.576959 26498 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.578171 26615 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.578913 26837 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.579262 26837 raft_consensus.cc:686] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:59.579758 26837 consensus_queue.cc:237] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.387099 27080 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:05:59.582762 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 85243.2us
I20240502 08:05:59.599758 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.600450 26570 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.603982 26691 tablet_bootstrap.cc:492] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64: No bootstrap required, opened a new log
I20240502 08:05:59.604473 26691 ts_tablet_manager.cc:1367] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64: Time spent bootstrapping tablet: real 0.175s user 0.096s sys 0.062s
I20240502 08:05:59.606957 26691 raft_consensus.cc:348] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.607661 26691 raft_consensus.cc:374] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:59.607925 26691 raft_consensus.cc:729] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Initialized, Role: FOLLOWER
I20240502 08:05:59.608634 26691 consensus_queue.cc:260] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.610806 26691 ts_tablet_manager.cc:1398] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:05:59.610616 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:59.611737 27063 raft_consensus.cc:2781] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.612779 27062 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.616384 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:59.617022 26642 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.620484 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.625536 26568 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:05:59.627391 26467 leader_election.cc:341] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:05:59.628835 26716 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.629285 26716 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.632202 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.632474 26716 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.632794 26501 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.633915 26615 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.634641 26837 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:59.635092 26837 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.635468 26837 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.636370 26568 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:59.638823 27063 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.639195 27063 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:59.640220 27109 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.640722 27109 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.642980 27109 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.651540 26570 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.653123 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.652271 27063 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.656445 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.657732 27063 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.658351 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:59.659453 26642 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:59.662078 26467 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:59.662745 27080 raft_consensus.cc:2781] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.662724 27068 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.663552 27080 raft_consensus.cc:686] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:59.664391 27080 consensus_queue.cc:237] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.665092 26837 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.667482 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.668097 26496 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.670246 26837 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.672250 26467 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:59.672073 26395 catalog_manager.cc:5484] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:59.672982 26982 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:05:59.673914 26395 catalog_manager.cc:5484] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:59.675206 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.675522 26395 catalog_manager.cc:5484] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:59.676558 26501 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.689433 27062 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.687216 26496 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.690122 27062 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.690970 26615 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.691664 26837 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:59.692150 27062 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.692320 26837 raft_consensus.cc:686] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:05:59.693353 26568 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.693807 26837 consensus_queue.cc:237] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.695875 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.696688 26643 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 1: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:59.696763 27062 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:05:59.699261 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.699962 27062 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:05:59.704699 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.704885 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.705379 26571 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 0.
I20240502 08:05:59.722461 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.727900 26541 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:05:59.731323 27109 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.737839 26395 catalog_manager.cc:5484] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:59.747795 27068 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:59.766404 27080 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20240502 08:05:59.800468 26395 catalog_manager.cc:5484] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:59.836297 27109 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.836788 27109 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.838675 27109 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.839957 26498 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:59.846837 26920 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.848623 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.849112 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.852928 26568 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 1: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:59.863070 26893 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.866466 26893 consensus_queue.cc:1035] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.871685 26893 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:59.886379 26920 consensus_queue.cc:1035] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.890940 26467 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:59.891983 26982 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:05:59.890956 26893 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.893528 26893 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.895437 26716 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:05:59.910298 26982 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:59.910781 26982 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.912657 26982 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.916996 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:05:59.917625 26643 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:59.919010 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:05:59.920727 27080 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:05:59.921087 27080 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.921433 27080 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:59.923346 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.923983 26571 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:05:59.929662 26893 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.930414 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:05:59.932473 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:05:59.933049 26576 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:05:59.934885 27080 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.935503 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:05:59.936167 26498 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:59.937031 27079 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:05:59.937402 27079 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:59.937731 27079 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:59.938272 27080 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:05:59.939900 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.940490 26568 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:59.944155 27079 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.943861 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:05:59.946439 26643 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:05:59.951326 27079 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:05:59.951908 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:05:59.952749 26501 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:05:59.952567 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:05:59.956069 26615 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:05:59.956591 26568 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:05:59.956872 26920 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:05:59.957942 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:05:59.958741 26982 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:59.959192 26982 raft_consensus.cc:686] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:05:59.959995 26982 consensus_queue.cc:237] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:05:59.970885 26399 catalog_manager.cc:5484] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 0 to 2, leader changed from <none> to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:00.073678 26893 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.083483 26920 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.136363 26982 consensus_queue.cc:1035] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.152823 26568 raft_consensus.cc:1226] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:00.154958 27109 consensus_queue.cc:1046] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.155953 26885 raft_consensus.cc:3032] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:00.156294 26885 raft_consensus.cc:729] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:00.157017 26885 consensus_queue.cc:260] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:00.158255 26885 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:00.195385 26920 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.203967 26568 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:06:00.211004 26837 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.381322 26576 raft_consensus.cc:1261] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:06:00.382428 26885 consensus_queue.cc:1035] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:00.395238 26643 raft_consensus.cc:1261] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:06:00.396814 27080 consensus_queue.cc:1035] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:01.090852 25352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:06:01.120218 27133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:06:01.121342 27134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:06:01.125941 25352 server_base.cc:1034] running on GCE node
W20240502 08:06:01.131448 27136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:06:01.132478 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:06:01.132710 25352 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:06:01.132902 25352 hybrid_clock.cc:648] HybridClock initialized: now 1714637161132885 us; error 0 us; skew 500 ppm
I20240502 08:06:01.133474 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:06:01.136262 25352 webserver.cc:460] Webserver started at http://127.24.194.4:37651/ using document root <none> and password file <none>
I20240502 08:06:01.136744 25352 fs_manager.cc:359] Metadata directory not provided
I20240502 08:06:01.136925 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:06:01.137226 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:06:01.138397 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-3-root/instance:
uuid: "5216c4a3a33f47ac88fa35704487f77e"
format_stamp: "Formatted at 2024-05-02 08:06:01 on dist-test-slave-753s"
I20240502 08:06:01.144480 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:06:01.148237 27141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:06:01.149212 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:06:01.149577 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-3-root
uuid: "5216c4a3a33f47ac88fa35704487f77e"
format_stamp: "Formatted at 2024-05-02 08:06:01 on dist-test-slave-753s"
I20240502 08:06:01.149947 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:06:01.172189 25352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:06:01.174115 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:06:01.176201 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:06:01.179466 25352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:06:01.179672 25352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:06:01.179919 25352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:06:01.180147 25352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:06:01.291832 25352 rpc_server.cc:307] RPC server started. Bound to: 127.24.194.4:45711
I20240502 08:06:01.292475 27204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.194.4:45711 every 8 connection(s)
I20240502 08:06:01.296468 25352 auto_leader_rebalancer-test.cc:405] add a tserver: 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:01.306854 27205 heartbeater.cc:347] Connected to a master server at 127.24.194.62:42981
I20240502 08:06:01.307248 27205 heartbeater.cc:464] Registering TS with master...
I20240502 08:06:01.307920 27205 heartbeater.cc:511] Master 127.24.194.62:42981 requested a full tablet report, sending...
I20240502 08:06:01.309574 26395 ts_manager.cc:194] Registered new tserver with Master: 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:01.311131 26395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52756
I20240502 08:06:01.653340 26716 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:01.654362 26716 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:01.660679 26716 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:01.662573 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:01.663478 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:01.663816 26500 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:01.664328 26568 raft_consensus.cc:2379] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:01.666319 26615 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:01.667757 26837 raft_consensus.cc:2781] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:01.668191 26837 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:01.668592 26837 raft_consensus.cc:3037] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:01.684110 26837 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:01.685935 26837 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:01.686658 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:01.687197 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:01.688031 26569 raft_consensus.cc:2379] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:01.693553 26501 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:01.694957 26615 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:01.695616 26716 raft_consensus.cc:2781] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:01.696151 26716 raft_consensus.cc:686] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:01.697281 26716 consensus_queue.cc:237] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:01.716779 26395 catalog_manager.cc:5484] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 2, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:01.810855 27146 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:01.811287 27146 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:01.813258 27146 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:01.814582 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:01.814877 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:01.815797 26501 raft_consensus.cc:2396] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:06:01.816694 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:01.817516 27146 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:02.078958 26569 raft_consensus.cc:1261] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:02.080209 26716 consensus_queue.cc:1035] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:06:02.091367 26500 raft_consensus.cc:1261] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:02.093567 27213 consensus_queue.cc:1035] T 24dcc168bb1b4064b7e7801a4f94f5ff P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:02.313546 27205 heartbeater.cc:503] Master 127.24.194.62:42981 was elected leader, sending a full tablet report...
I20240502 08:06:02.347576 26568 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.354382 26643 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.356369 27146 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:02.363034 26500 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.367884 27146 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.369236 27063 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 08:06:02.376224 26541 consensus_peers.cc:485] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc069ce0343a4e9b9056d3604d2928df. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:02.376657 26643 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.379685 26500 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.392068 26399 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 1 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:02.419879 26500 consensus_queue.cc:237] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.426549 26643 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.426803 26568 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.429105 27080 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:02.430230 27212 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:06:02.440936 26466 consensus_peers.cc:485] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a26db3828484577a5da525732f0b102. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:02.443039 27212 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.444442 26643 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.445508 26568 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.456862 26569 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.461278 26396 catalog_manager.cc:5484] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:02.469086 26642 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:02.470695 27146 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:06:02.472712 26541 consensus_peers.cc:485] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d53d3e99887443b8967129ddd219819. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:02.474592 26500 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:02.475942 27146 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:02.488245 27227 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.498564 26643 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.496812 26500 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.513828 26396 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:02.562484 26642 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.568321 26569 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.569897 26837 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:02.572062 26501 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:02.573684 27213 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:06:02.590008 26615 consensus_peers.cc:485] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ab970a1718349829f0f76fa212d5594. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:02.591603 26837 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.607844 26570 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.613794 26501 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.627208 26399 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:02.759116 27250 ts_tablet_manager.cc:902] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:02.768013 27250 tablet_copy_client.cc:308] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.2:40601
I20240502 08:06:02.783532 26586 tablet_copy_service.cc:140] P 97dc9560dbbe4d1aa44b2b2edda4080f: Received BeginTabletCopySession request for tablet fc069ce0343a4e9b9056d3604d2928df from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:43852)
I20240502 08:06:02.784222 26586 tablet_copy_service.cc:161] P 97dc9560dbbe4d1aa44b2b2edda4080f: Beginning new tablet copy session on tablet fc069ce0343a4e9b9056d3604d2928df from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:43852: session id = 5216c4a3a33f47ac88fa35704487f77e-fc069ce0343a4e9b9056d3604d2928df
I20240502 08:06:02.792145 26586 tablet_copy_source_session.cc:205] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:02.795490 27250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc069ce0343a4e9b9056d3604d2928df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:02.807910 27250 tablet_copy_client.cc:783] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:02.808604 27250 tablet_copy_client.cc:647] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:02.814030 27256 ts_tablet_manager.cc:902] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:02.823673 27250 tablet_copy_client.cc:523] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:02.825611 27256 tablet_copy_client.cc:308] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.1:40577
I20240502 08:06:02.835283 27250 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:02.836625 26511 tablet_copy_service.cc:140] P 8f9fe1f3763547388c4301b13a127aaa: Received BeginTabletCopySession request for tablet 9a26db3828484577a5da525732f0b102 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:58270)
I20240502 08:06:02.837128 26511 tablet_copy_service.cc:161] P 8f9fe1f3763547388c4301b13a127aaa: Beginning new tablet copy session on tablet 9a26db3828484577a5da525732f0b102 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:58270: session id = 5216c4a3a33f47ac88fa35704487f77e-9a26db3828484577a5da525732f0b102
I20240502 08:06:02.845031 26511 tablet_copy_source_session.cc:205] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:02.848084 27256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a26db3828484577a5da525732f0b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:02.866179 27256 tablet_copy_client.cc:783] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:02.865020 27250 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:02.866931 27256 tablet_copy_client.cc:647] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:02.867944 27250 tablet_bootstrap.cc:492] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:02.868733 27250 ts_tablet_manager.cc:1367] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.004s
I20240502 08:06:02.871917 27250 raft_consensus.cc:348] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.872876 27250 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:02.874259 27256 tablet_copy_client.cc:523] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:02.873533 27250 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.881029 27263 ts_tablet_manager.cc:902] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:02.884531 27256 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:02.899986 27250 ts_tablet_manager.cc:1398] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.031s user 0.014s sys 0.000s
I20240502 08:06:02.903345 26586 tablet_copy_service.cc:342] P 97dc9560dbbe4d1aa44b2b2edda4080f: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-fc069ce0343a4e9b9056d3604d2928df received from {username='slave'} at 127.0.0.1:43852
I20240502 08:06:02.903784 26586 tablet_copy_service.cc:433] P 97dc9560dbbe4d1aa44b2b2edda4080f: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-fc069ce0343a4e9b9056d3604d2928df on tablet fc069ce0343a4e9b9056d3604d2928df with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:02.936662 27256 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:02.937507 27256 tablet_bootstrap.cc:492] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:02.938184 27256 ts_tablet_manager.cc:1367] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.054s user 0.030s sys 0.014s
I20240502 08:06:02.940207 27256 raft_consensus.cc:348] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.940922 27256 raft_consensus.cc:729] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:02.941526 27256 consensus_queue.cc:260] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:02.943572 27263 tablet_copy_client.cc:308] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.2:40601
I20240502 08:06:02.943996 27256 ts_tablet_manager.cc:1398] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 08:06:02.945700 26586 tablet_copy_service.cc:140] P 97dc9560dbbe4d1aa44b2b2edda4080f: Received BeginTabletCopySession request for tablet 8d53d3e99887443b8967129ddd219819 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:43852)
I20240502 08:06:02.946118 26511 tablet_copy_service.cc:342] P 8f9fe1f3763547388c4301b13a127aaa: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-9a26db3828484577a5da525732f0b102 received from {username='slave'} at 127.0.0.1:58270
I20240502 08:06:02.946280 26586 tablet_copy_service.cc:161] P 97dc9560dbbe4d1aa44b2b2edda4080f: Beginning new tablet copy session on tablet 8d53d3e99887443b8967129ddd219819 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:43852: session id = 5216c4a3a33f47ac88fa35704487f77e-8d53d3e99887443b8967129ddd219819
I20240502 08:06:02.946519 26511 tablet_copy_service.cc:433] P 8f9fe1f3763547388c4301b13a127aaa: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-9a26db3828484577a5da525732f0b102 on tablet 9a26db3828484577a5da525732f0b102 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:02.963812 26586 tablet_copy_source_session.cc:205] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:02.966958 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d53d3e99887443b8967129ddd219819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:02.979321 27263 tablet_copy_client.cc:783] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:02.979864 27263 tablet_copy_client.cc:647] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:02.982936 27263 tablet_copy_client.cc:523] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:02.988905 27263 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:03.019272 27263 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:03.020102 27263 tablet_bootstrap.cc:492] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:03.020604 27263 ts_tablet_manager.cc:1367] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.014s
I20240502 08:06:03.023667 27263 raft_consensus.cc:348] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:03.024528 27263 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:03.025131 27263 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:03.027335 27263 ts_tablet_manager.cc:1398] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240502 08:06:03.028837 26586 tablet_copy_service.cc:342] P 97dc9560dbbe4d1aa44b2b2edda4080f: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-8d53d3e99887443b8967129ddd219819 received from {username='slave'} at 127.0.0.1:43852
I20240502 08:06:03.029230 26586 tablet_copy_service.cc:433] P 97dc9560dbbe4d1aa44b2b2edda4080f: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-8d53d3e99887443b8967129ddd219819 on tablet 8d53d3e99887443b8967129ddd219819 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:03.150666 27263 ts_tablet_manager.cc:902] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:03.170691 27263 tablet_copy_client.cc:308] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.3:40473
I20240502 08:06:03.186223 26660 tablet_copy_service.cc:140] P 755f35bc66a841f382d1c4df6c6f6b64: Received BeginTabletCopySession request for tablet 3ab970a1718349829f0f76fa212d5594 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:54230)
I20240502 08:06:03.186790 26660 tablet_copy_service.cc:161] P 755f35bc66a841f382d1c4df6c6f6b64: Beginning new tablet copy session on tablet 3ab970a1718349829f0f76fa212d5594 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:54230: session id = 5216c4a3a33f47ac88fa35704487f77e-3ab970a1718349829f0f76fa212d5594
I20240502 08:06:03.197927 26660 tablet_copy_source_session.cc:205] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:03.200834 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab970a1718349829f0f76fa212d5594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:03.215579 27263 tablet_copy_client.cc:783] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:03.216301 27263 tablet_copy_client.cc:647] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:03.219843 27263 tablet_copy_client.cc:523] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:03.226967 27180 raft_consensus.cc:1203] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:06:03.227991 27263 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:03.253556 27263 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:03.254349 27263 tablet_bootstrap.cc:492] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:03.255039 27263 ts_tablet_manager.cc:1367] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.012s
I20240502 08:06:03.257424 27263 raft_consensus.cc:348] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:03.258306 27263 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:03.258916 27263 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:03.260962 27263 ts_tablet_manager.cc:1398] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:06:03.262499 26660 tablet_copy_service.cc:342] P 755f35bc66a841f382d1c4df6c6f6b64: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-3ab970a1718349829f0f76fa212d5594 received from {username='slave'} at 127.0.0.1:54230
I20240502 08:06:03.262863 26660 tablet_copy_service.cc:433] P 755f35bc66a841f382d1c4df6c6f6b64: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-3ab970a1718349829f0f76fa212d5594 on tablet 3ab970a1718349829f0f76fa212d5594 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:03.280275 27180 raft_consensus.cc:1203] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 08:06:03.315883 27180 raft_consensus.cc:1203] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:06:04.742805 27276 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:04.743383 27276 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:04.747798 27080 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:04.748234 27080 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:04.748657 27276 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:04.749743 27080 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.059329 27276 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.062577 27276 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.064666 27276 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.065081 27080 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.067633 27080 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.069973 27080 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.070760 27276 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.071857 27276 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.074594 27277 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.072126 27280 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.075116 27277 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.075244 27280 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.077260 27277 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.077404 27280 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.079913 27080 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.077915 27277 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.080602 27080 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.081254 27277 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.083870 27279 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.084415 27279 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.074705 27276 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.086357 27080 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.089293 27080 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.089816 27080 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.091022 27276 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.086463 27280 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.092216 27282 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.092443 27280 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.091286 27278 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.092559 27276 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.094424 27280 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.094558 27278 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.085846 27283 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.095964 27283 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.096706 27278 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.097329 27276 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.098150 27280 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.099210 27280 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.092217 27080 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.098503 27276 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.101507 27276 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.105245 27276 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.097931 27283 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.088521 27281 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.103545 27280 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.092877 27282 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.109269 27281 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.111727 27281 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.112205 27282 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.088399 27279 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.113157 27279 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.113811 27279 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.116335 27279 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.116659 27286 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.117187 27286 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.120096 27276 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.121158 27276 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.126140 27280 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.126818 27280 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.133761 27280 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.136152 27280 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.137502 27280 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.141182 27280 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.097349 27278 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.149920 27278 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.152221 27278 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.124862 27276 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.156078 27277 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.162194 27213 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.162761 27213 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.166062 27296 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.166594 27296 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.167565 27295 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.168105 27295 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.169142 27296 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.174523 27296 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.176616 27296 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.169063 27294 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.178694 27294 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.182353 27294 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.179616 27296 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.126215 27283 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.183437 27294 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.183980 27294 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.167420 27213 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.136662 27284 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.185129 27284 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.186460 27294 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.183756 27283 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.180308 27295 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.182804 27299 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.186677 27297 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.196071 27284 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.198951 27293 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.199126 27287 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.203119 27293 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.203724 27289 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.206574 27289 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.207726 27293 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.208602 27289 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.230769 27080 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.231386 27080 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.231832 27288 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.232421 27288 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.203204 27287 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.237517 27080 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.237846 27283 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.238584 27287 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.243672 27300 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.243889 27287 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.244438 27287 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.244375 27300 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.246582 27287 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.247069 27287 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.247526 27287 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.249750 27287 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.251405 27292 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.253315 27292 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.255553 27292 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.201954 27299 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.240576 27280 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.261579 27284 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.268810 27285 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.205615 27297 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.275307 27283 raft_consensus.cc:1050] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:05.267374 27279 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.270660 27281 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.271618 27080 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.271211 27276 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.274878 27300 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.274945 27286 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.241214 27289 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.282099 27080 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.280946 27293 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.280797 27276 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:05.276394 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 59176.6us
I20240502 08:06:05.279021 27281 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.287423 27301 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.287671 27297 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.278167 27284 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.283360 27285 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.272540 27280 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.289530 27281 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.280824 27288 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.284091 27289 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.290601 27293 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.279150 27279 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.291839 27080 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.290170 27300 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.287900 27301 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.301133 27308 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.297536 27310 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.298197 27286 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.300475 27283 consensus_queue.cc:237] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:05.299342 27276 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.297524 27312 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.311439 27290 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.300881 27302 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.312299 27289 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.315606 27296 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.316329 27298 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.311967 27290 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.317472 27285 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.318648 27279 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.302826 27280 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.318461 27302 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.322155 27293 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.322628 27299 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.301678 27308 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.319208 27284 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.338382 27301 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:05.305809 27300 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.319437 27287 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.328871 27313 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.313212 27286 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.334441 27312 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.343233 27296 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.313889 27310 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.359738 27307 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.349370 27298 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.365957 27298 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.367966 27302 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.370815 27290 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.372751 27296 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.361339 27299 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.374678 27299 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.354562 27287 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.375335 27286 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.417150 27309 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.414451 27308 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.377466 27295 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.369330 27294 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.360544 27307 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.400758 27301 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.417735 27309 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.429128 27298 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.404790 27318 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.428644 27321 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.429354 27294 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:05.426909 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 140919us
I20240502 08:06:05.433806 27298 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.438144 27317 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.441916 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.438302 27316 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.428277 27295 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.437278 27213 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.449542 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.436445 27318 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.434615 27301 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.468595 27304 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.469053 27298 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.442849 27303 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.430980 27315 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.446271 27308 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.446417 27286 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.470830 27213 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.471953 27300 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.480619 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.361433 27313 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.467249 27292 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.467512 27290 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.469745 27297 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.484031 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.513857 26642 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.482450 27309 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.469156 27304 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.482398 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.474838 27291 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.439977 27321 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.527174 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:05.560993 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 93159.1us
I20240502 08:06:05.481738 27286 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.565935 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.568568 27320 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.570703 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.526202 27290 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.571550 26642 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:05.482662 27308 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.573035 26643 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.517892 27287 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.560171 27299 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.528230 27310 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.449661 27317 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.528420 27311 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.518147 27213 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.516727 27305 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:05.590183 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.590868 26497 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:05.582450 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.591313 27180 raft_consensus.cc:1203] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:06:05.577891 27290 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.575125 27307 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.582257 27312 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.582701 27323 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.581331 27313 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.584002 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.528165 26645 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:05.602243 26500 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:05.570683 27293 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.583693 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.476768 27316 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.517036 27292 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.594870 27322 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.517517 27297 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.587549 27294 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.582890 27295 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.608543 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.579658 27320 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.606457 27318 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.603423 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.600647 27307 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.562920 27291 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.602406 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.511875 27315 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.606230 27322 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.511198 27303 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.756451 27286 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.756762 26648 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.757131 27286 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.636516 27327 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.604940 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.625274 27293 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.770833 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.762468 26496 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:05.770260 27307 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.625521 27301 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.661612 27297 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.637354 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.759747 27321 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.775923 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.782128 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.782861 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.784217 26648 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.784533 27317 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.757643 27286 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.784906 27293 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.594965 27311 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.788738 27322 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.762220 27327 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.788877 27286 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.604264 27305 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.607079 27323 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.826079 27316 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.798208 27308 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.842928 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:05.812371 27304 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.846382 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.847246 26642 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:05.812783 27292 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.841272 27322 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.812556 27286 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.812906 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.856767 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.864871 27320 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.865483 27327 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.852921 27322 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.866160 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.866667 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.858776 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:05.877431 26569 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:05.879177 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.879436 27322 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.879982 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.882787 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.883576 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.895300 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.853894 27286 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.911259 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.911607 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.912475 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.913163 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.913306 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.913632 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.907516 27303 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.926623 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.928418 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.928933 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.914680 27307 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.919235 27311 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.931717 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.918005 27322 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.935468 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.921542 26648 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.937815 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.937963 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.948091 27305 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.938714 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:05.951946 27323 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.947726 27322 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.946839 27303 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.953693 27303 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.937815 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.933046 27291 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:05.962527 27322 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.963399 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.968366 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.968716 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.969336 26492 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:05.962534 27286 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.970054 26501 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:05.974941 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.975185 27328 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.976233 27323 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.976192 27328 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.976799 27323 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.977871 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.978547 27323 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.980209 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.982553 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.982915 27327 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.983497 27286 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.983453 27327 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:05.984030 27286 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.984736 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.985514 27327 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.986455 27329 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.986778 27322 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.987293 27322 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.987653 27307 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.988760 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.989825 27320 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.990357 27320 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.985791 26647 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:05.987020 27329 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.989171 27322 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.988751 27307 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.974956 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:05.994376 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:05.996059 27323 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:05.996594 27323 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.999064 27292 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.009011 27304 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.009541 27304 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.010207 27308 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.010813 27308 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.009352 27292 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.978466 27328 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:05.986024 27327 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:05.985790 27286 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.051703 27179 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.051846 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:06.082962 27201 debug-util.cc:398] Leaking SignalData structure 0x7b080025f300 after lost signal to thread 25355
I20240502 08:06:06.054978 27327 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:05.992236 27315 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.112279 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.073438 26498 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:06.076822 27283 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240502 08:06:06.120108 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.120664 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.122339 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.123396 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.123598 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.123708 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.112895 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:06.121903 27201 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf4c0 after lost signal to thread 26433
I20240502 08:06:06.129037 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.124683 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.120021 27329 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.123600 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.133109 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.131166 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.126610 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.134096 26571 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.132599 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.134419 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.131673 26499 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:06.133222 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.133109 27327 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.128384 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.138058 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.138401 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.138772 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.139442 26571 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.138427 26494 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:06.140906 27307 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.140942 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.141325 27304 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.141829 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:06.142416 27201 debug-util.cc:398] Leaking SignalData structure 0x7b08001c4fc0 after lost signal to thread 26525
W20240502 08:06:06.143172 27201 debug-util.cc:398] Leaking SignalData structure 0x7b08002e7a20 after lost signal to thread 26600
I20240502 08:06:06.142979 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.143533 27308 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:06.143920 27201 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec060 after lost signal to thread 26674
I20240502 08:06:06.144153 27303 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.144564 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.145386 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.145658 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.147400 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.147734 27320 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.148492 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.149613 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.150753 26571 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.152665 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.153362 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.154235 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.144914 27292 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.154547 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:06.144948 27201 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec8e0 after lost signal to thread 27204
I20240502 08:06:06.149721 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.159022 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.159215 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.160182 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.160267 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.154465 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.149070 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.165345 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.166504 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.166610 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.169059 26649 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.169188 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.170118 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.170886 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.165904 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.162963 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.163259 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.163364 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.174722 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.159262 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.165199 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.171558 26574 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.173702 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.173864 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.174315 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.181908 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.182214 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.182591 26572 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.176393 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.174824 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.175290 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.177542 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.185365 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.185803 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.186336 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.187544 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.188158 26574 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.185941 26568 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.189858 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.191224 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.191262 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.194249 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.195211 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.195312 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.195839 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.196611 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.198200 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.175302 27323 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.198858 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.178232 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.199785 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.176992 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.180002 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.182821 26567 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.184109 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.201747 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.202327 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.205370 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.205636 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.207167 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.207432 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.207782 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.208774 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.209960 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.210582 26567 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.210754 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.212754 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.212808 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.213105 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.214346 26569 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.214826 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.216030 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.216238 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.217933 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.218465 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.219094 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.220340 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.220474 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.222499 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.224653 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:06.225078 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.225885 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.226110 26567 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:06.226244 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.227046 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.227645 26542 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.227810 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.225102 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.228969 26542 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.229337 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.229532 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.230525 27303 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.230933 27303 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.231299 27303 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.232858 27320 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.233268 27320 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.233654 27320 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.236863 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.237501 26500 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:06.238653 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.242417 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.242930 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.242484 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.243223 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.244143 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.244449 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.245294 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.245741 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.245890 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.246953 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.248170 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.249567 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.247169 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.247397 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.244458 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.247702 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.245888 26575 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:06.254395 26542 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.255573 27292 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.255959 27292 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.256363 27292 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.257902 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.259020 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.259455 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.260298 26542 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.261225 27308 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.261664 27308 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.262190 27308 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.262463 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.263935 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.268186 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.268962 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:06.270108 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.270746 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.271400 26615 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:06.271967 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.273523 26615 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:06.272361 27291 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.272732 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.274212 27291 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.274605 27291 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.274945 26542 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.274961 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.275617 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.276518 27323 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.276906 27323 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.277263 27323 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.277315 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.279088 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.279413 27317 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.279770 27317 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.280141 27317 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.281093 26542 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.282078 27287 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.282467 27287 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.282817 27287 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.286754 27317 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.288736 27317 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.302891 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.303067 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.304795 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.307524 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.311056 26615 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:06.314181 27307 raft_consensus.cc:1050] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:06.314878 26616 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.318075 27309 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.317202 27329 raft_consensus.cc:1050] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:06.316635 27307 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:06.321938 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.325937 27303 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.326444 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.328557 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.320677 27329 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:06.330150 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.330952 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.331126 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.331763 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.332109 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.316933 27287 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.322841 27317 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.333573 27317 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.329443 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.334223 27317 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.328763 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.334484 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.335484 27309 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.337109 26616 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.330076 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.330332 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.338248 27287 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.333551 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.339417 26615 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.329339 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.339648 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.340641 27316 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.341193 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.323786 27291 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:06.342604 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:06.343364 27291 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.343703 26615 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.294806 27292 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.344952 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.345551 26569 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.346158 27292 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.344615 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.346812 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.341395 27295 raft_consensus.cc:2726] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.348183 26541 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.349514 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.350003 26541 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.350747 27304 raft_consensus.cc:2726] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.351795 27287 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.347047 27320 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.304319 27323 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.354883 26541 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.356881 27292 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.357214 26615 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.354542 27316 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.358501 27316 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.358762 26615 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:06.359722 27301 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.360168 27301 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.360600 27301 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.362758 26615 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.327982 27303 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.362920 27317 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.370668 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.373517 27323 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.383590 27316 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.384074 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.385483 26615 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.386242 27316 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.386543 27295 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.386663 27316 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.385146 27283 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.387573 27283 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.388046 26642 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:06:06.388085 27283 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.354997 27308 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.390468 27327 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:06.391106 27320 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.394455 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.394616 27301 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.389729 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.396625 27305 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.396732 26650 raft_consensus.cc:1226] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.397610 26615 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.398586 26616 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.398808 26615 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.396346 26541 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.400293 26615 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.401499 26615 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.402544 26644 raft_consensus.cc:1226] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.402736 27329 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.403530 27301 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.405496 26541 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.406234 26615 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.406934 27305 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.407233 27329 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.407545 27305 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.407593 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.408042 27305 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.407264 26615 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.406955 26541 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.410030 26615 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.408409 27305 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.409092 27309 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.415329 27309 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.415681 27309 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.417026 26642 raft_consensus.cc:1226] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.417568 26541 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.408304 27308 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.418884 26541 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.404383 27291 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.418529 27311 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.420596 27311 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.420673 27291 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.404227 27318 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.421056 27311 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.418440 27327 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.420686 27295 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.423579 26541 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.423139 27307 consensus_queue.cc:1046] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:06.425017 27327 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.426121 27327 raft_consensus.cc:3032] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.426451 27327 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.427254 27327 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:06.422914 27295 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:06.430806 27295 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.435062 27317 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.429808 27327 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.418394 27320 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.422991 27329 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.439258 27287 consensus_queue.cc:1046] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240502 08:06:06.447013 27287 raft_consensus.cc:3032] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.447391 27287 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.448174 27287 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.449486 27287 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.451102 27309 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.437816 27303 consensus_queue.cc:1046] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240502 08:06:06.453774 27329 raft_consensus.cc:3032] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.454172 27329 raft_consensus.cc:729] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.454999 27329 consensus_queue.cc:260] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.455729 27283 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.456390 27329 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:06.457788 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 48756.2us
I20240502 08:06:06.453186 27309 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
W20240502 08:06:06.460719 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62212.8us
I20240502 08:06:06.462754 27295 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.460243 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:06.460484 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 50519us
I20240502 08:06:06.441388 26541 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.466789 26615 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.466989 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.467527 27340 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.467691 27335 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.467996 27340 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.468144 27335 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.468428 27340 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.468538 27303 raft_consensus.cc:2726] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.468575 27335 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.469575 26541 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.470434 26467 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.471041 27303 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.471371 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.472664 26467 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.473706 27316 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.474334 27295 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.474812 26645 raft_consensus.cc:1226] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.474979 27333 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.476030 27333 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.476430 27333 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.475313 27334 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.475651 27316 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.457644 27311 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.477896 27317 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.479301 26615 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.482945 27305 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.483680 27311 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.485469 27305 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.485407 27335 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:06.486985 27283 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.487746 27283 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.488199 27283 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.490648 27283 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.493067 27333 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.495126 27333 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.479910 27334 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.480399 26541 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.488989 27317 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.496590 27340 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.481266 27307 consensus_queue.cc:1046] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.503562 26645 raft_consensus.cc:1226] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.503724 27307 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.503849 26541 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.504455 27307 raft_consensus.cc:3032] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.504805 27307 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.505116 26541 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.506086 27323 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.506197 27320 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.506465 26541 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.507560 27323 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.507802 26541 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.508105 27323 consensus_queue.cc:1046] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.505609 27307 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.509593 27307 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.498517 27335 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.500324 27283 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.502377 27334 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.511605 26615 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.512063 27323 raft_consensus.cc:3032] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.511603 27308 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.512454 27323 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.514972 26541 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.514977 26615 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.518304 26541 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.517349 27305 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.518895 27311 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.519444 26541 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.520861 26615 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.521049 26541 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.521472 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.521958 26541 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.521901 26615 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.522311 26467 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.520262 27327 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.520344 27320 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.524315 27315 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.523548 27335 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.524323 27333 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.525175 27335 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.525393 27333 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.525633 27335 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.525815 27333 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.518430 27303 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.527244 27311 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.528416 26615 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.529120 27318 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.529599 27318 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.531193 27333 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.531648 26615 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.532398 27311 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.532763 27311 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.532860 27333 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.533164 27311 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.520359 27340 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.513456 27323 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.524016 27329 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.535158 27323 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.534010 27335 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.544203 27335 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.545029 26467 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.547039 26467 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.539139 27334 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.549477 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.547569 27335 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.548238 27343 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.550202 27333 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.539673 26541 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.551379 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:06.539794 26615 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.552635 27343 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.572269 27341 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.572608 27329 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.574322 27333 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.574720 27334 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.573969 27343 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.574872 27333 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.540247 27283 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.577185 26615 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.552217 27335 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.573427 26541 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.584525 27341 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.585417 26541 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.586313 27329 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.586658 26541 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.587103 27329 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.584645 27335 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.539278 27311 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.590384 27307 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.592435 27311 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:06.584514 27283 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.593410 27283 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:06.593890 27283 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.596215 26541 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.598843 27329 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.599987 26541 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.600654 27329 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.601714 26541 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.602512 27329 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.603565 26541 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.605851 27329 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.606859 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.607662 27329 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.611418 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.612654 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.613904 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.614601 27283 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.616526 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.617755 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.618108 27283 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.618921 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.620169 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.621078 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:06.621299 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 69971.9us
I20240502 08:06:06.622289 26498 raft_consensus.cc:1226] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.623533 26541 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.623646 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.624663 27329 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.624624 26541 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.606757 27343 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.625895 26500 raft_consensus.cc:1226] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.626819 27333 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.627010 27343 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.628110 27329 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.628677 27333 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.629441 26541 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.630164 27329 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.630545 26541 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.631227 27329 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.632280 26541 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.632647 27311 raft_consensus.cc:1050] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:06.632959 27329 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.636176 26496 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.636495 26541 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.637499 26541 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.637758 26500 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.635149 27311 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:06.638670 27335 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.638983 27343 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:06.640530 27335 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.643121 27323 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.643608 26499 raft_consensus.cc:1226] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.643122 26541 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.644733 27341 consensus_queue.cc:1046] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.644587 27323 raft_consensus.cc:2726] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.644579 27329 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.645725 27305 raft_consensus.cc:3032] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.646140 27305 raft_consensus.cc:729] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.647127 27317 consensus_queue.cc:1046] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.646955 27305 consensus_queue.cc:260] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.648313 27341 raft_consensus.cc:3032] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.648389 27305 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.648837 27341 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.642547 26467 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.648221 27317 consensus_queue.cc:1046] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.651095 27342 consensus_queue.cc:1046] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.652074 27342 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.652405 27342 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.651345 27318 raft_consensus.cc:3032] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.653191 27318 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.654233 27318 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.653254 27342 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.656924 27342 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.657903 26541 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.659098 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.652616 27317 consensus_queue.cc:1046] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.660801 27343 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.661016 27318 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.661231 27343 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.664695 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.665231 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.665614 26569 raft_consensus.cc:2396] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:06.664350 27344 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:06.666885 27344 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.667277 27344 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:06.666898 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.661674 27343 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.673861 27179 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:06:06.677848 26645 raft_consensus.cc:1226] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.667191 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.680253 27180 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.680562 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.681365 26650 raft_consensus.cc:1226] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.682363 26643 raft_consensus.cc:1226] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.664930 27329 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.686414 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.686596 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.687784 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.688524 27323 consensus_queue.cc:1046] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.689599 27323 consensus_queue.cc:1046] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:06.690685 27287 consensus_queue.cc:1046] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.691253 27323 raft_consensus.cc:3032] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.691573 27315 raft_consensus.cc:3032] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.691979 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.692211 27315 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.649633 27341 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.695176 26568 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:06.697824 26570 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:06.697973 26499 raft_consensus.cc:1226] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.691685 27323 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.691690 27287 raft_consensus.cc:3032] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.701805 26616 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.702517 27341 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.703476 27307 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:06.705747 27180 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.708860 27287 raft_consensus.cc:729] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:06.712080 27323 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.711244 27316 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.712512 27287 consensus_queue.cc:260] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.713661 27323 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.713920 27287 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.714637 27291 raft_consensus.cc:3032] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.715018 27291 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.715773 27291 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.716881 27291 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.715804 27316 consensus_queue.cc:1046] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240502 08:06:06.720064 26575 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:06.720793 27343 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.717522 27309 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240502 08:06:06.726183 27343 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.693025 27315 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:06.728127 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 62329.1us
I20240502 08:06:06.730005 26467 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.720710 27316 raft_consensus.cc:3032] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.730795 27316 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.730837 27334 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.731766 27316 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.733081 27316 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.734210 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.734962 27334 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:06.735301 27334 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:06.735728 27334 raft_consensus.cc:3037] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.739988 26499 raft_consensus.cc:1226] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.743543 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.745162 26467 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.746573 26467 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.747960 26467 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.742827 27309 consensus_queue.cc:1046] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:06.748478 27345 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.748870 27343 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.756537 26467 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:06.756882 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 55156.1us
I20240502 08:06:06.757853 26616 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.758968 27295 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.755862 27335 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.759783 27301 raft_consensus.cc:3032] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.760138 27301 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.760797 26616 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.756057 27333 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.761503 26499 raft_consensus.cc:1226] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.761746 27340 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.762143 27295 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.763460 26616 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.761547 27301 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:06.765432 27301 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.763154 27295 consensus_queue.cc:1046] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240502 08:06:06.767374 27309 raft_consensus.cc:3032] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.767707 27309 raft_consensus.cc:729] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.766160 27311 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.764695 26467 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.768450 27309 consensus_queue.cc:260] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.765436 26616 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.775130 26467 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.776296 27311 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.776679 27309 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.774061 27283 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.782203 27343 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:06.765550 27315 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.775930 26467 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.785277 26467 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.776288 27334 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.787004 26616 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.786329 27340 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.769053 27344 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:06.789585 26467 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.790380 27345 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.790488 27344 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.791134 27334 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:06.791424 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.792093 27334 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.793700 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.794375 27343 raft_consensus.cc:2726] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.785310 27283 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.798468 26616 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.798254 27318 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.800060 27318 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:06.800575 27318 raft_consensus.cc:686] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.801854 27318 consensus_queue.cc:237] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.802398 26616 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.803148 27348 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:06.804006 27348 raft_consensus.cc:686] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.800771 26467 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.805532 26467 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.806622 26467 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.807456 27343 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.806669 27348 consensus_queue.cc:237] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.808657 26467 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.809384 27343 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.810324 26467 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.810910 27343 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.806730 27334 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.811630 26616 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.806056 27283 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.812453 27305 raft_consensus.cc:2781] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:06.813573 26467 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.816874 27287 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:06.818660 27343 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.819555 26568 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.815186 27305 raft_consensus.cc:686] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.820763 26467 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.822947 27305 consensus_queue.cc:237] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.825863 26496 raft_consensus.cc:1226] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.827025 27344 raft_consensus.cc:2726] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.827498 27342 consensus_queue.cc:1046] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.828534 26467 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.823815 26499 raft_consensus.cc:1226] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.829344 27344 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.828955 27316 raft_consensus.cc:3032] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.830448 27316 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.830857 26467 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.831233 27316 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.842837 27316 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.843246 26396 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:06.814067 27283 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:06.834820 26645 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:06.831591 27180 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:06.839723 27342 consensus_queue.cc:1046] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.850745 27342 raft_consensus.cc:3032] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.851140 27342 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.851900 27342 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.852594 26541 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.852929 27342 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.853677 27323 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.854454 27283 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.855521 26541 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.856560 27323 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.858909 26569 raft_consensus.cc:1226] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.861030 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.872380 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.884996 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.893775 26499 raft_consensus.cc:1226] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.896180 27316 consensus_queue.cc:1046] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.895328 26399 catalog_manager.cc:5484] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 2, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:06.898834 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.902678 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.902940 26645 raft_consensus.cc:2396] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:06.899856 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.903836 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.908102 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.910326 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.914455 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.914860 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.917469 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.902009 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.902170 27342 raft_consensus.cc:3032] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.917611 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.918311 27342 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.919286 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.919175 27342 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.920367 27342 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.900717 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.922925 26648 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.924861 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.926541 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.927183 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.929178 26570 raft_consensus.cc:1226] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.929286 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.930500 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.932338 26541 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:06.933318 27292 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.933388 26500 raft_consensus.cc:1226] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:06.934417 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.934540 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.935570 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:06.936475 26642 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.936532 26498 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:06:06.939693 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.964578 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.965497 26568 raft_consensus.cc:1226] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.928586 26646 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.943259 27343 consensus_queue.cc:1046] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.966786 27343 raft_consensus.cc:3032] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.967155 27343 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:06.968020 27343 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.969427 27343 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.970736 26498 raft_consensus.cc:1226] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.971269 26647 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.972489 26645 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.972493 27342 consensus_queue.cc:1046] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:06.973649 27342 raft_consensus.cc:3032] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:06.973975 27342 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:06.974944 27342 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.976281 27342 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:06.944079 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.979058 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:06.945101 26641 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.945885 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.979801 27283 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.946112 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.980489 27283 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:06.947911 26573 raft_consensus.cc:1226] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.948552 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.949272 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.986817 26572 raft_consensus.cc:2379] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:06.950388 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.950465 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:06.955355 26569 raft_consensus.cc:1226] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.994185 26645 raft_consensus.cc:1226] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:06.998191 27292 consensus_queue.cc:1046] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:06.999365 26574 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.959139 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.960186 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:06.999390 27292 raft_consensus.cc:3032] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.005487 27292 raft_consensus.cc:729] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:07.008215 27292 consensus_queue.cc:260] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.009428 27292 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.018383 27344 consensus_queue.cc:1046] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.019404 27345 raft_consensus.cc:3032] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.019774 27345 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.020604 27345 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:06.962795 26643 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.963711 26644 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.957458 26650 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:06.982189 27283 consensus_queue.cc:1046] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240502 08:06:07.028306 27334 raft_consensus.cc:3032] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.028702 27334 raft_consensus.cc:729] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.029517 27334 consensus_queue.cc:260] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.030795 27334 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.035617 26395 catalog_manager.cc:5484] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 2, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:07.037319 26395 catalog_manager.cc:5484] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 2, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:07.048126 27345 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.056840 27347 consensus_queue.cc:1046] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:07.057288 27283 consensus_queue.cc:1046] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.058234 26467 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:07.058899 27333 raft_consensus.cc:3032] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.059063 27347 raft_consensus.cc:3032] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.059360 27333 raft_consensus.cc:729] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.059525 27347 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.060241 27333 consensus_queue.cc:260] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.060460 27347 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.061623 27333 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.061908 27347 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.062302 27343 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.064410 27343 raft_consensus.cc:686] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.065378 27343 consensus_queue.cc:237] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.079522 27343 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:07.090548 26541 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:07.091706 27292 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:07.093225 26541 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:07.094553 27292 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:07.098085 27292 consensus_queue.cc:1046] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:07.099109 27292 raft_consensus.cc:3032] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:07.100855 27180 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:07.105831 26576 raft_consensus.cc:1226] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:07.107451 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.108281 27347 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:07.099449 27292 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:07.109604 26467 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.110560 26541 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:07.111027 27347 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.109370 27292 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:07.114183 27292 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:07.111835 27347 raft_consensus.cc:686] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.112234 27323 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:07.115286 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.118649 27345 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.119185 27345 raft_consensus.cc:686] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.119956 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.120116 27345 consensus_queue.cc:237] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.120759 27334 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:07.122206 26467 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.123008 27334 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.122323 26395 catalog_manager.cc:5484] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:07.123553 27334 raft_consensus.cc:686] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.124521 27334 consensus_queue.cc:237] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.119158 27347 consensus_queue.cc:237] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.133442 26467 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.135458 27343 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.135987 27343 raft_consensus.cc:686] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.136744 26467 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.139576 27343 consensus_queue.cc:237] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.141255 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.141880 27347 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.143025 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.143669 27347 raft_consensus.cc:686] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.143894 27345 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:07.143692 27334 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.144711 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.145457 27345 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.145900 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.145928 27345 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.146917 27333 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.147017 26467 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:07.147109 27345 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.150198 27335 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:07.150662 27335 raft_consensus.cc:686] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.147961 27347 consensus_queue.cc:237] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.148633 27334 raft_consensus.cc:686] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.151484 27335 consensus_queue.cc:237] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.149892 27333 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.154928 27333 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.168764 26644 raft_consensus.cc:1226] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:07.172385 26650 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:07.173002 26574 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:07.182852 27283 consensus_queue.cc:1046] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.184041 27340 raft_consensus.cc:3032] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.184805 27340 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.185550 27340 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.186698 27340 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.186520 27334 consensus_queue.cc:237] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.195053 27344 consensus_queue.cc:1046] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240502 08:06:07.210717 26567 raft_consensus.cc:1261] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.212217 27348 consensus_queue.cc:1035] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.218183 27333 raft_consensus.cc:3032] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:07.220379 27333 raft_consensus.cc:729] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:07.223853 26498 raft_consensus.cc:1261] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.225008 27316 consensus_queue.cc:1035] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.242376 27333 consensus_queue.cc:260] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:07.256012 26576 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.250870 27333 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.259357 27342 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.492173 26567 raft_consensus.cc:1261] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 08:06:07.497095 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 174339us
I20240502 08:06:07.503666 26496 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.505072 26650 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.507941 27335 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.508762 27342 consensus_queue.cc:1035] T 77b2256f7eb5420792d8042fb8e30d0b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:07.508828 26395 catalog_manager.cc:5484] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.511185 26498 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.517437 26395 catalog_manager.cc:5484] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.519392 26395 catalog_manager.cc:5484] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.520872 26395 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.522411 26395 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.523861 26395 catalog_manager.cc:5484] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.524621 26496 raft_consensus.cc:1261] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.525444 26395 catalog_manager.cc:5484] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.527086 26395 catalog_manager.cc:5484] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.528762 27316 consensus_queue.cc:1035] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.528666 26395 catalog_manager.cc:5484] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.530891 26395 catalog_manager.cc:5484] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:07.535470 26498 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.545024 27316 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.598853 26644 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.599566 26643 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.600925 26569 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.612066 26569 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.614234 26643 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.614512 26569 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.618575 27369 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.625834 27369 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.627537 27369 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.628782 27345 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:07.632741 26569 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.631371 26496 consensus_queue.cc:237] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:07.634883 26567 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.635586 26574 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.636901 26643 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.639433 27345 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:07.678234 26575 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.643785 26572 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.644654 26573 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.645516 26571 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.655436 26650 raft_consensus.cc:1261] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.656047 26569 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.707284 27345 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.708699 26574 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.709967 27334 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.711665 26567 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.711854 27334 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.717548 27373 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.718919 27373 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.720489 27180 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:06:07.722491 26643 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:06:07.717847 26571 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.723920 26643 raft_consensus.cc:1261] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.727430 26569 raft_consensus.cc:1261] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.736948 26567 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:07.741053 27369 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:07.742913 27369 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:07.747467 27369 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
W20240502 08:06:07.904299 27389 log.cc:924] Time spent T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20240502 08:06:07.750908 27335 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.146s
I20240502 08:06:07.782749 27347 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
W20240502 08:06:07.899277 27392 log.cc:924] Time spent T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20240502 08:06:07.915817 26573 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.954883 26575 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:07.975987 26567 raft_consensus.cc:1261] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:08.309964 27400 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:08.376456 27400 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:08.380724 27400 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.020781 27405 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.021543 27405 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.024511 26539 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.004s sys 0.005s
W20240502 08:06:09.024984 26539 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.712s user 0.004s sys 0.005s
W20240502 08:06:09.044314 26430 debug-util.cc:398] Leaking SignalData structure 0x7b0800039be0 after lost signal to thread 25355
W20240502 08:06:09.133126 25357 debug-util.cc:398] Leaking SignalData structure 0x7b080000ec40 after lost signal to thread 26615
W20240502 08:06:09.133796 25357 kernel_stack_watchdog.cc:198] Thread 26615 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 300ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20240502 08:06:09.202967 26542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.005s sys 0.008s
W20240502 08:06:09.203620 26542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.005s sys 0.008s
I20240502 08:06:09.364055 27405 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.364418 27407 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.365111 27407 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.368866 27409 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.369400 27409 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.365839 27405 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.390957 27408 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.391435 27408 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.393432 27408 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.390424 27406 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.396004 27409 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.396549 27409 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.396270 27406 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.397133 27409 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.398823 27406 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.398957 27409 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.399456 27406 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.400024 27409 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.400523 27409 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.400315 27406 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.394567 27405 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.393927 27408 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.403465 27411 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.402895 27408 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.404696 27410 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.404974 27407 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.405246 27410 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.405998 27412 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.407716 27412 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.408313 27406 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.410106 27412 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.405689 27411 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.408221 27410 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.404156 27409 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.412912 27408 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.413221 27411 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.405545 27407 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.414456 27407 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.416002 27406 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.416704 27406 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.419397 27411 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.417743 27412 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.419925 27411 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.420123 27412 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.424914 27409 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.425410 27409 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.427248 27408 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.427446 27410 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.427903 27410 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.427747 27408 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.431383 27413 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.431835 27413 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.435467 27414 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.436060 27414 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.438223 27414 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.438678 27414 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.438925 27413 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.439164 27414 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.441308 27414 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.439426 27413 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.441855 27414 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.441936 27413 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.442438 27414 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.443905 27413 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.444154 27414 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.445782 27415 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.446303 27415 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.448043 27415 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:09.450706 26574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.545s user 0.005s sys 0.015s
W20240502 08:06:09.451211 26574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.546s user 0.005s sys 0.016s
I20240502 08:06:09.451999 26574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 509) took 1685 ms. Trace:
I20240502 08:06:09.452739 26568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 508) took 1729 ms. Trace:
I20240502 08:06:09.452343 26574 rpcz_store.cc:276] 0502 08:06:07.766371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.782785 (+ 16414us) service_pool.cc:225] Handling call
0502 08:06:07.904280 (+121495us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:07.904638 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:07.904674 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:07.905146 (+ 472us) log.cc:842] Serialized 26 byte log entry
0502 08:06:09.451618 (+1546472us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:09.451660 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:09.451734 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:09.451796 (+ 62us) raft_consensus.cc:1671] finished
0502 08:06:09.451811 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:09.451926 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2864,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":4612,"thread_start_us":1546108,"threads_started":1,"wal-append.queue_time_us":407902,"wal-append.run_cpu_time_us":1956,"wal-append.run_wall_time_us":1034480}
I20240502 08:06:09.453153 26568 rpcz_store.cc:276] 0502 08:06:07.723205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.723335 (+ 130us) service_pool.cc:225] Handling call
0502 08:06:07.733741 (+ 10406us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:07.734030 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:07.734077 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:07.734463 (+ 386us) log.cc:842] Serialized 26 byte log entry
0502 08:06:07.930808 (+196345us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:07.930846 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:07.930906 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:09.452512 (+1521606us) raft_consensus.cc:1671] finished
0502 08:06:09.452533 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:09.452679 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1413,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":183212,"threads_started":1,"wal-append.queue_time_us":530137,"wal-append.run_cpu_time_us":2440,"wal-append.run_wall_time_us":1055166}
I20240502 08:06:09.458065 27415 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.458204 27414 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.458621 27415 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.458662 27414 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.460652 26466 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.550s user 0.011s sys 0.014s
I20240502 08:06:09.460913 27414 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:09.461253 26466 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.553s user 0.011s sys 0.014s
I20240502 08:06:09.461642 27414 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.462548 27414 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.465909 27414 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.466499 27414 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.466974 27414 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.467411 27415 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.468494 27415 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.469108 27414 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.469264 27340 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.470022 27343 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.469693 27415 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.470644 27343 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.469811 27340 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.472673 27343 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.473210 27343 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.473641 27343 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.474027 27340 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.475556 27343 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.470610 27417 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.476485 27417 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.469693 27414 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.476485 27407 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.485723 27408 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.485548 27414 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.486095 27417 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.481580 27413 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.483069 27412 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.487301 27412 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.487484 27407 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.487797 27412 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.488039 27407 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.489359 27412 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.489567 27333 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.484495 27415 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.489969 27412 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.490090 27333 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.491235 27418 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.490803 27412 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.491721 27418 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.492322 27333 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.490191 27407 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.493033 27412 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.486687 27408 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.487223 27413 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.499043 27413 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.499704 27413 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.500216 27413 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.501381 27414 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.502352 27414 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.502677 27413 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.496846 27418 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.494417 27417 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.503417 27414 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.504674 27417 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.509413 26615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.678s user 0.007s sys 0.023s
I20240502 08:06:09.514549 27417 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:09.517788 26570 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.556s user 0.011s sys 0.002s
I20240502 08:06:09.496040 27415 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:09.518173 26570 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.556s user 0.012s sys 0.002s
I20240502 08:06:09.494241 27408 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.518319 27415 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.518954 26570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 503) took 1818 ms. Trace:
I20240502 08:06:09.519822 27408 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.519917 27415 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.515091 27414 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.497754 27412 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.493739 27333 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.520750 27412 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.494295 27416 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:09.517971 26615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.686s user 0.008s sys 0.024s
I20240502 08:06:09.523959 27412 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.520994 27333 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.494342 27407 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.526501 27407 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:09.527253 27422 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.528779 27407 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.527973 27422 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:09.530731 27333 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.503443 27413 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.531333 27422 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.531435 27413 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.533151 27422 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.534289 27422 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.533670 27413 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.538441 27413 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.538956 27413 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.540685 26467 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.646s user 0.006s sys 0.029s
W20240502 08:06:09.541141 26467 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.646s user 0.006s sys 0.029s
I20240502 08:06:09.519297 26570 rpcz_store.cc:276] 0502 08:06:07.700505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.700653 (+ 148us) service_pool.cc:225] Handling call
0502 08:06:07.701099 (+ 446us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:07.701555 (+ 456us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:07.701594 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:07.702170 (+ 576us) log.cc:842] Serialized 26 byte log entry
0502 08:06:07.961680 (+259510us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b1800139060. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011d3eb5 00007f970a00d939 00007f97011f142b 00007f97011ef82f 00007f97011f1992 00007f9703d9d4eb 00007f9703da2f93 00007f9703da3341 00007f97040f081f
0502 08:06:09.518554 (+1556874us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:09.518592 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:09.518651 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:09.518691 (+ 40us) raft_consensus.cc:1671] finished
0502 08:06:09.518703 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:09.518898 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"spinlock_wait_cycles":15726464,"thread_start_us":1556701,"threads_started":1,"wal-append.queue_time_us":1323473}
I20240502 08:06:09.540818 27416 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.543390 27416 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.541144 27413 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.548679 27333 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.549156 27333 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.537936 27422 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.551087 27333 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.551816 27345 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.552979 27369 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.552731 27345 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.553465 27369 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.555224 27345 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.555315 27369 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.555814 27345 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.556231 27369 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.556295 27345 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.556771 27369 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.558507 27345 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.558909 27369 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.559374 27418 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.560037 27418 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.561632 27343 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:09.563059 27418 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.559441 27369 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.563964 27369 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.566365 27369 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.567157 27418 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.569813 27418 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.571620 27348 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.571888 27418 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.572453 27418 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.572157 27348 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.572918 27418 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.574369 27348 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:09.568789 27340 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.574894 27418 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.574942 27340 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.575551 27418 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.576253 27418 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:09.576941 27340 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.577750 27342 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.578394 27418 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.567819 27345 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.569262 27333 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.575394 27283 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.578953 27422 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.579265 27333 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.579382 27283 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.579546 27422 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.581115 26572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 506) took 1858 ms. Trace:
I20240502 08:06:09.581388 27283 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.578970 27345 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.582743 27409 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.584185 27345 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.578243 27342 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.585590 27348 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.585630 27405 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.586551 27406 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.586833 27422 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.587920 27422 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.588490 27422 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.587430 27342 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.590361 27342 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.590864 27342 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.586581 27348 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.592783 27342 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.593362 27342 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:09.593670 26614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.735s user 0.001s sys 0.013s
I20240502 08:06:09.593816 27342 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.594920 26614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.737s user 0.002s sys 0.013s
I20240502 08:06:09.592783 27333 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.596453 27425 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.597088 27425 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.581543 26572 rpcz_store.cc:276] 0502 08:06:07.722385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.726989 (+ 4604us) service_pool.cc:225] Handling call
0502 08:06:09.524225 (+1797236us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b58001e0398. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f9709073060 00007f970905e5a9 00007f970905e052 00007f9702740694
0502 08:06:09.524430 (+ 205us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:09.524786 (+ 356us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:09.524824 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:09.525312 (+ 488us) log.cc:842] Serialized 26 byte log entry
0502 08:06:09.572470 (+ 47158us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:09.574469 (+ 1999us) log.cc:842] Serialized 14 byte log entry
0502 08:06:09.580697 (+ 6228us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:09.580775 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:09.580818 (+ 43us) raft_consensus.cc:1671] finished
0502 08:06:09.580927 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:09.581056 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5979,"prepare.queue_time_us":1516,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":2715,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":205,"spinlock_wait_cycles":8726784,"thread_start_us":48338,"threads_started":2,"wal-append.queue_time_us":45228}
I20240502 08:06:09.598987 27428 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.599223 27425 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.599717 27425 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.599550 27428 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.600180 27425 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.602447 27428 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.605641 27430 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.606216 27430 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.608603 27430 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.610189 27422 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.611207 27348 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.614406 27342 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.615796 27343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60736 (request call id 106) took 1986 ms. Trace:
W20240502 08:06:09.616680 27144 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.701s user 0.008s sys 0.024s
W20240502 08:06:09.617265 27144 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.703s user 0.009s sys 0.025s
I20240502 08:06:09.618153 27335 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.618434 27425 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.618914 27432 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.618784 27335 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:09.619352 27432 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.619432 27180 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:09.620913 27335 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.621264 27432 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.622296 27384 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:09.622874 26464 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.713s user 0.019s sys 0.020s
W20240502 08:06:09.623507 26464 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.721s user 0.019s sys 0.020s
I20240502 08:06:09.622946 27384 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.620357 27425 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:09.625099 26430 debug-util.cc:398] Leaking SignalData structure 0x7b080003e2c0 after lost signal to thread 26433
W20240502 08:06:09.625478 26430 debug-util.cc:398] Leaking SignalData structure 0x7b080003e380 after lost signal to thread 26447
I20240502 08:06:09.625957 27428 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.624339 27425 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.629045 27418 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.630632 27333 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.630843 27425 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.631150 27345 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.631875 27340 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.631749 27345 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.632364 27340 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.633596 27345 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.634317 27348 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.634788 27348 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.631171 27333 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.636462 27410 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.636585 27348 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.637106 27348 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.637486 27411 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.638391 27342 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.629627 27418 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.639791 26465 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.745s user 0.013s sys 0.017s
W20240502 08:06:09.640293 26465 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.745s user 0.013s sys 0.017s
I20240502 08:06:09.641170 27342 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.642326 26430 debug-util.cc:398] Leaking SignalData structure 0x7b08002f43a0 after lost signal to thread 26525
W20240502 08:06:09.642761 26430 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4360 after lost signal to thread 26526
I20240502 08:06:09.621749 27432 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.625399 27384 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.626545 27428 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.644302 27432 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:09.643997 26430 debug-util.cc:398] Leaking SignalData structure 0x7b080032c340 after lost signal to thread 26600
I20240502 08:06:09.645752 27428 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.646095 27432 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:09.646292 26430 debug-util.cc:398] Leaking SignalData structure 0x7b0800338c40 after lost signal to thread 26674
W20240502 08:06:09.647509 26430 debug-util.cc:398] Leaking SignalData structure 0x7b08002abe00 after lost signal to thread 27204
I20240502 08:06:09.647764 27342 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:09.647950 26430 debug-util.cc:398] Leaking SignalData structure 0x7b080014a780 after lost signal to thread 27339
W20240502 08:06:09.648389 26430 debug-util.cc:398] Leaking SignalData structure 0x7b080020de00 after lost signal to thread 27350
I20240502 08:06:09.648483 26399 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39264 (request call id 126) took 1978 ms. Trace:
I20240502 08:06:09.649055 26399 rpcz_store.cc:276] 0502 08:06:07.669809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.669959 (+ 150us) service_pool.cc:225] Handling call
0502 08:06:09.648384 (+1978425us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1834,"reported_tablets":4}
I20240502 08:06:09.650630 27283 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.651260 27283 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.616225 27343 rpcz_store.cc:276] 0502 08:06:07.629215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:07.629338 (+ 123us) service_pool.cc:225] Handling call
0502 08:06:07.669809 (+ 40471us) log.cc:842] Serialized 494 byte log entry
0502 08:06:09.615704 (+1945895us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35968,"raft.queue_time_us":46162,"raft.run_cpu_time_us":938,"raft.run_wall_time_us":937}
I20240502 08:06:09.667671 27345 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.668254 27345 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.670333 27345 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.670805 27345 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.671227 27345 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.672180 27369 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.672991 27345 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.673419 27333 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.673524 27345 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:09.688252 27335 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.702556 27428 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.702503 27335 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.703299 27428 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.704420 27436 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.675612 27340 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.705740 27428 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.674046 27333 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:09.693341 27283 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.707391 27436 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.709386 27436 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.712474 27411 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.675522 27434 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.685809 27418 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.692690 27342 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.713452 27419 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.713160 27411 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.713603 27342 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.713939 27419 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.716003 27419 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.716145 27342 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.716523 27411 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.716835 27434 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.717514 27411 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.718207 27411 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.719069 27434 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.719724 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.720012 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.720291 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.720398 26644 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.720932 26648 raft_consensus.cc:2396] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:09.721108 26649 raft_consensus.cc:2396] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:09.721940 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.722458 27411 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.722560 26650 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.704689 27335 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.676633 27373 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.705075 27426 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.723672 27426 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.725204 27411 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.637495 27348 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.672639 27369 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.726222 27410 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.684276 27422 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.726751 27373 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:09.712823 27431 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.726770 27342 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:09.728776 27369 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.728717 27384 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.729207 27342 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.725616 27426 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.729547 27384 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.729574 27410 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.729235 27431 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.731683 27384 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.732039 27410 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.732348 27384 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.732553 27410 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.728293 27432 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.733047 27410 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.733497 27432 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.734386 27373 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.735538 27432 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.735810 27410 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.736085 27432 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.725726 27411 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.736591 27432 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.738574 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.738358 27422 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.731675 27342 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.739276 26642 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:09.745066 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.726332 27421 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.747779 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.749919 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.748335 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.746145 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.752352 26642 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.753520 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.753604 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.753685 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.754141 27425 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.754473 26649 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.754547 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.755470 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.755600 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.755926 27373 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.756207 26646 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.756453 27373 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.748982 27342 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.757114 27424 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.757668 27342 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.757761 27424 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.758797 26644 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.756774 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:09.750615 26643 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
W20240502 08:06:09.745551 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 73857.9us
I20240502 08:06:09.760643 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.763753 26496 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:09.761904 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.739131 27411 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.744465 27410 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.762455 27426 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:09.754215 26641 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.766368 26498 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:09.754653 27425 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.740604 27430 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.732899 27431 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.745293 27428 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.771490 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.775306 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.780171 26501 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.780578 26643 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.781364 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.765880 27342 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.778296 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.759236 27373 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.775542 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.781383 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.790416 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.785105 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.793237 26573 raft_consensus.cc:2396] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:09.781920 26643 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.793637 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.780078 27430 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.794287 26641 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.794472 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.795183 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.795600 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.786314 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.797119 27425 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.797191 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.797822 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.797916 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.789938 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.798794 26570 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 1.
I20240502 08:06:09.799047 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.799175 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.791010 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.767241 26645 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.792444 26647 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.792582 26576 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.800354 26650 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.803982 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.803346 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.797986 26494 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.799964 26571 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:09.732838 27384 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.776670 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.812407 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.813117 26498 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.807113 26642 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.815650 26647 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:09.816417 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.816704 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.817060 26500 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.792733 27348 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.795545 27430 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.768879 27434 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240502 08:06:09.820904 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.819612 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.821753 26643 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.819825 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.822155 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.822474 26501 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.822799 26568 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.822984 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.823295 27439 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.823659 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.823832 27439 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.824738 26644 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:09.818653 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.827104 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.828496 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.829140 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.829255 26501 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.829633 26573 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.795794 26641 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.769428 27410 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.803014 27411 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.770813 27426 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.812368 27425 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.812180 27424 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.800364 27419 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.834019 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.811942 27432 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.833735 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:09.842566 27433 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.842674 27420 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.812731 27342 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.768054 27421 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.802213 26649 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.788646 27428 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.843017 27179 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.844921 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.846777 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.848935 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.849583 26650 raft_consensus.cc:2379] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:09.830945 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.831992 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.853935 26501 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.855466 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.856393 26385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a26db3828484577a5da525732f0b102 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 08:06:09.857372 26541 service_pool.cc:135] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:51122 dropped due to backpressure. The service queue is full; it has 50 items.
I20240502 08:06:09.843431 27342 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.860369 26650 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:09.843578 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:09.840194 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.861810 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.843202 27433 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.856931 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.857501 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.864619 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:09.872602 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.873204 26501 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.866087 27384 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.838616 27411 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.854733 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.877313 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.841758 27425 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.880563 26501 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.861196 27435 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.883989 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.857241 27421 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.843219 27420 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:09.892267 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.893889 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.894611 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.894861 27428 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.894986 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.895411 27411 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.895669 26494 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.895772 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.896423 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.870257 27439 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.853174 27419 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.866977 27178 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.875380 26572 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.883517 27422 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.867743 26541 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:51122 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 523)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 659)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 511)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 514)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 679)
I20240502 08:06:09.905344 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.905985 26576 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.908550 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.908833 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.909072 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.909592 26496 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.909847 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.909900 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.910984 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.911561 26568 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:09.911625 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.912699 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.912794 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.913497 26575 raft_consensus.cc:2396] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:09.913461 26498 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.915102 26570 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.918520 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.919131 26496 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.919853 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.923640 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.925531 26498 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.929052 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.903762 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:09.930368 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.930464 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.929716 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.931208 26571 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.931553 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.871548 26540 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:51146 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 523)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 659)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 510)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 511)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 512)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 513)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 514)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 516)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 521)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 679)
I20240502 08:06:09.932034 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.932173 26648 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.932703 26571 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.933126 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.933511 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.934514 27433 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:09.934916 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.935536 26568 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.935801 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.936462 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.937073 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.937469 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.937712 27426 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.935452 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.937842 26499 raft_consensus.cc:2379] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:09.939054 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.941090 26495 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.933676 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.942088 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.942377 26572 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.893890 27420 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:09.943485 26568 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.939550 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.930619 27179 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.934446 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.955976 26569 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.946650 26571 raft_consensus.cc:2379] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:09.958014 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.958717 26568 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.914501 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.959344 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.959628 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.959985 26573 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:09.960259 26571 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.960698 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.961135 26573 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.959120 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.961063 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.961980 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.962461 26571 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:09.962711 26573 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.963266 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.964005 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.964279 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.964998 26571 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:09.964928 26541 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.961920 26569 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:09.966223 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.966645 26541 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.967043 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.967542 26569 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.968196 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.968338 26541 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.968709 26569 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:09.969278 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.964444 26573 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.969645 26541 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.969663 26569 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.970423 26541 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.967386 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.943245 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.973431 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.970554 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.932030 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.970317 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.975881 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.943241 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.988540 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.977358 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.989418 26642 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.940364 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.938470 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.978255 27323 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:09.972996 26644 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:09.978709 26541 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.973276 27406 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:09.978888 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.005004 26541 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.972966 27405 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:09.947480 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.006153 27406 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:09.939309 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.007272 27413 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.007678 27413 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:09.943241 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.980078 26572 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.980371 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.007746 26567 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.984817 27410 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.009502 26647 raft_consensus.cc:2396] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:10.009898 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.010334 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.973134 27411 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:09.981758 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.011945 26542 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:09.982987 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.017522 26576 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:09.939218 26492 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:09.977684 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.020761 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.976414 26569 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:10.023162 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.974444 26571 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.990682 26616 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:09.990918 26499 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:09.984392 27419 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:09.990919 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.008649 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.020223 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.008299 27342 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:10.018821 27411 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.032267 26572 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:10.032591 27411 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.033136 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.990638 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.034291 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.034826 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.035367 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.035872 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.036542 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.037029 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.037178 26572 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:09.900622 27435 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:09.994406 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:09.987260 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.999809 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.004618 27420 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.020823 27416 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:10.006271 26541 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.011047 26647 raft_consensus.cc:2379] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:10.009006 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.016906 26570 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:10.060609 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:09.983933 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.018060 26542 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.016585 27406 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:09.991626 27323 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.083521 27323 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.083714 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.002015 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.008204 27405 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.024689 26616 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.008062 27413 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.026825 26568 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:09.992663 26648 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.027087 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.028313 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.030308 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.117146 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.117635 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.045394 27420 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.118013 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.118434 27420 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.118841 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.117146 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.030395 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.042237 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.008390 27425 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:10.020081 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:09.973124 27409 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.043566 26494 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.020081 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.082892 27421 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:10.085167 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 67207.5us
W20240502 08:06:10.085069 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 78873.7us
I20240502 08:06:10.086493 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:10.087901 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 63283.8us
I20240502 08:06:10.113500 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.115599 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.115846 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.116366 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.120146 26576 raft_consensus.cc:2396] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:10.134032 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.156850 26642 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.040385 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.161551 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.207075 26568 raft_consensus.cc:2379] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:10.119272 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.174150 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.234952 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.112916 27419 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:10.112574 27405 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.196161 26616 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.195720 26570 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
W20240502 08:06:10.241546 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 45469.3us
W20240502 08:06:10.242605 26616 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:51146 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 3 similar messages]
I20240502 08:06:10.174228 27409 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:10.242707 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.243199 27409 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.120451 26649 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.177177 26542 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.117798 26641 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.129151 27410 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.092808 26646 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
I20240502 08:06:10.235015 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.235015 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.236140 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.250164 26642 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:10.250218 26576 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:10.194522 26492 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:10.251119 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.251348 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.251195 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.252549 26576 raft_consensus.cc:2396] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240502 08:06:10.252580 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.245105 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.198015 26541 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:10.246373 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 69325.2us
I20240502 08:06:10.248670 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:10.253949 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 56053.2us
I20240502 08:06:10.237138 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.156540 27342 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.253437 26568 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.200738 26497 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.254850 26647 raft_consensus.cc:2379] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:10.254807 26541 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.254945 26542 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.258875 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.265866 26650 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.254175 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.244117 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.267261 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.270793 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.270951 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.271425 26642 raft_consensus.cc:2379] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:10.271477 26497 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:10.269933 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.272432 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.272655 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.273032 26497 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.249404 27419 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.279642 27419 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.280020 27419 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.280370 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.271968 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.240722 26572 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.286721 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.287346 26572 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.287832 27283 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:10.281193 27436 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240502 08:06:10.202616 27411 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.293594 27411 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.264290 27416 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.295864 27416 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:10.296241 27416 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.294966 26569 raft_consensus.cc:2354] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:10.245597 27425 raft_consensus.cc:2781] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.298306 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.294204 27411 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.247530 27410 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.259280 27342 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.284786 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.285286 26501 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:10.315659 26395 catalog_manager.cc:5484] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 3 to 4, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New cstate: current_term: 1 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:10.330551 27452 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:06:10.329874 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 87284.6us
I20240502 08:06:10.350446 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.772454 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.773129 26644 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.329495 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.778662 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.779457 26646 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:10.780406 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.303100 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.336227 26649 raft_consensus.cc:2396] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:10.819304 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.820516 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.277230 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.822108 26650 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:10.822957 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.824132 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.824728 26650 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.457741 27145 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.442857 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.832222 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.833003 26643 raft_consensus.cc:2379] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
W20240502 08:06:10.835490 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 62654.4us
I20240502 08:06:10.351466 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.398319 27323 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.317220 27342 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.267547 27373 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240502 08:06:10.396340 27406 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.307502 27410 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.859723 27406 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.403331 27405 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.396682 27409 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.304386 27411 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:10.865098 27411 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.866308 26576 raft_consensus.cc:1226] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:10.403331 27413 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.398742 27419 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.396893 27416 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.879030 27419 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.879643 27416 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.405650 27420 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.300056 27425 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:10.891023 27425 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.275795 26492 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.562212 27435 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.892357 27435 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.249219 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.892560 27420 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.892714 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.892994 26494 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:10.893612 26492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 320) took 1019 ms. Trace:
W20240502 08:06:10.598312 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 319939us
I20240502 08:06:10.626909 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:10.630831 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 376048us
W20240502 08:06:10.891252 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 433602us
I20240502 08:06:10.899504 26541 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.900231 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.648784 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.901054 26496 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:10.901279 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.687377 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:10.679946 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 425025us
I20240502 08:06:10.677727 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.713969 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:10.909752 27426 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.910079 26575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 682) took 1073 ms. Trace:
I20240502 08:06:10.717341 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.745859 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.911247 27343 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.911391 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 319) took 1048 ms. Trace:
I20240502 08:06:10.911852 26499 rpcz_store.cc:276] 0502 08:06:09.863290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.745553 (+882263us) service_pool.cc:225] Handling call
0502 08:06:10.911332 (+165779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.912832 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.912951 26467 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.747675 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.768806 27421 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.770948 26616 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.781574 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.782657 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.790297 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.808655 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.811650 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.919752 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.915856 27421 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.920419 26493 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.921330 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.817520 26641 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
I20240502 08:06:10.893987 26492 rpcz_store.cc:276] 0502 08:06:09.874207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.892440 (+1018233us) service_pool.cc:225] Handling call
0502 08:06:10.893563 (+ 1123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.825721 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.923190 26650 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:10.854293 26649 raft_consensus.cc:2396] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:10.857519 27323 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.861310 26573 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:10.863070 27405 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.865751 27409 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.872018 26576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 667) took 1061 ms. Trace:
I20240502 08:06:10.927456 26576 rpcz_store.cc:276] 0502 08:06:09.808026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.301947 (+493921us) service_pool.cc:225] Handling call
0502 08:06:10.304323 (+ 2376us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:10.866790 (+562467us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:10.869188 (+ 2398us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b5400373f54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:10.869363 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1291867392}
I20240502 08:06:10.874991 26574 raft_consensus.cc:2379] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:10.876048 27413 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.879664 26568 raft_consensus.cc:1226] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240502 08:06:10.885902 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 71203.6us
I20240502 08:06:10.887342 26571 raft_consensus.cc:2379] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:10.893873 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.901499 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 302) took 1064 ms. Trace:
I20240502 08:06:10.932714 26496 rpcz_store.cc:276] 0502 08:06:09.837025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:09.923981 (+ 86956us) service_pool.cc:225] Handling call
0502 08:06:10.648602 (+724621us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800005198. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:10.901452 (+252850us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9413504}
I20240502 08:06:10.669837 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.907436 26541 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.908492 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:10.910177 27426 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.910748 26500 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:10.916041 26467 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.935408 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.937502 26467 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.914026 26498 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
W20240502 08:06:10.915216 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 144353us
I20240502 08:06:10.939476 26616 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.892717 27435 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:10.918334 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 680) took 1060 ms. Trace:
I20240502 08:06:10.914311 27420 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.911727 27343 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.855916 26567 raft_consensus.cc:2354] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:10.910527 26575 rpcz_store.cc:276] 0502 08:06:09.836722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.713632 (+876910us) service_pool.cc:225] Handling call
0502 08:06:10.909998 (+196366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.941555 26573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 687) took 1100 ms. Trace:
I20240502 08:06:10.941220 26501 rpcz_store.cc:276] 0502 08:06:09.857587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.808343 (+950756us) service_pool.cc:225] Handling call
0502 08:06:10.918278 (+109935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.917335 27436 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.917956 27428 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:10.926870 26644 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.944136 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 315) took 1088 ms. Trace:
I20240502 08:06:10.966265 26500 rpcz_store.cc:276] 0502 08:06:09.846173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:09.847387 (+ 1214us) service_pool.cc:225] Handling call
0502 08:06:10.717116 (+869729us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800005198. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:10.934637 (+217521us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8615680}
I20240502 08:06:10.946713 26542 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.968922 26542 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.948025 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.922765 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.945181 27439 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.952023 27413 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.977672 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.957881 26573 rpcz_store.cc:276] 0502 08:06:09.841234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:09.978676 (+137442us) service_pool.cc:225] Handling call
0502 08:06:10.941360 (+962684us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54001bba54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:10.941485 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1967853440}
I20240502 08:06:10.958477 26541 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.960084 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.943188 26467 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:10.954105 26568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 512) took 1153 ms. Trace:
I20240502 08:06:10.941193 27426 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.970779 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.944861 27435 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.980644 26492 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.990774 26492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 303) took 1153 ms. Trace:
I20240502 08:06:10.991108 26492 rpcz_store.cc:276] 0502 08:06:09.837190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.922505 (+1085315us) service_pool.cc:225] Handling call
0502 08:06:10.990711 (+ 68206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.992048 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.964855 27428 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:10.992978 27428 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:10.993209 27418 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.945039 27434 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.994225 27434 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.994494 27418 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.994652 27434 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.994930 27418 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.944703 27333 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.011873 27333 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.012426 27333 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.975337 27409 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.015229 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.977257 26644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 327) took 1058 ms. Trace:
I20240502 08:06:11.017527 26644 rpcz_store.cc:276] 0502 08:06:09.917040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.773765 (+856725us) service_pool.cc:225] Handling call
0502 08:06:10.975481 (+201716us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:11.021180 27463 log.cc:924] Time spent T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240502 08:06:10.952906 26571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 688) took 1103 ms. Trace:
I20240502 08:06:10.983345 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:11.026228 26571 rpcz_store.cc:276] 0502 08:06:09.849996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.116139 (+266143us) service_pool.cc:225] Handling call
0502 08:06:10.952670 (+836531us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b540023f854. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:10.952838 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1591070976}
I20240502 08:06:10.987143 27411 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.044708 27411 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:10.979652 26568 rpcz_store.cc:276] 0502 08:06:09.801147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.279432 (+478285us) service_pool.cc:225] Handling call
0502 08:06:10.283907 (+ 4475us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:10.930626 (+646719us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:10.953783 (+ 23157us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54002a1054. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:10.954013 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1369572608}
I20240502 08:06:10.974189 26542 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.944228 27420 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.072548 27420 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.942413 26649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 333) took 1014 ms. Trace:
I20240502 08:06:10.942802 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.976454 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.957557 27343 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.921936 26493 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:10.950322 27145 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:10.983803 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:10.984391 27405 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.985168 27465 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:10.985884 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:11.078284 26649 rpcz_store.cc:276] 0502 08:06:09.918023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.820259 (+902236us) service_pool.cc:225] Handling call
0502 08:06:10.932464 (+112205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.080736 27465 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.078944 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 694) took 1091 ms. Trace:
I20240502 08:06:11.081615 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:11.082873 26495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 314) took 1237 ms. Trace:
I20240502 08:06:10.962860 27436 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.977864 27413 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.088163 27436 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.088295 27413 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:10.989145 26572 raft_consensus.cc:1226] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:11.093174 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:11.093940 26572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 660) took 1292 ms. Trace:
I20240502 08:06:11.094064 26498 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:10.997730 26541 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:10.998311 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:11.094866 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 704) took 1011 ms. Trace:
I20240502 08:06:11.095752 27369 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:11.095659 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 97427.4us
I20240502 08:06:11.084901 27405 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.001793 27435 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:11.096869 27405 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:11.098297 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 100680us
I20240502 08:06:11.098680 27435 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.002259 27426 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:10.958698 27342 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.100402 27426 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.100610 27409 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.101037 27409 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.101418 27409 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.102025 26497 raft_consensus.cc:2354] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:11.103073 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 312) took 1258 ms. Trace:
I20240502 08:06:11.079527 26496 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:11.002256 27410 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.081164 27465 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.081559 26499 rpcz_store.cc:276] 0502 08:06:09.987731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.915215 (+927484us) service_pool.cc:225] Handling call
0502 08:06:11.078847 (+163632us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:10.975888 27439 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:10.988134 26574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 520) took 1174 ms. Trace:
I20240502 08:06:11.083318 26495 rpcz_store.cc:276] 0502 08:06:09.845427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.081316 (+1235889us) service_pool.cc:225] Handling call
0502 08:06:11.082801 (+ 1485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.085553 27411 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.085688 26493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 288) took 1258 ms. Trace:
W20240502 08:06:11.086903 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 159091us
I20240502 08:06:11.087589 27466 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.096347 27369 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.100404 27342 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.110512 26650 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240502 08:06:11.111250 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:11.079790 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 105735us
I20240502 08:06:11.113263 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 318) took 1250 ms. Trace:
I20240502 08:06:11.113929 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:11.152508 26568 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:11.114063 26467 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.116523 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:11.103430 26497 rpcz_store.cc:276] 0502 08:06:09.844882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.273667 (+428785us) service_pool.cc:225] Handling call
0502 08:06:11.102881 (+829214us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54000fb4d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:11.103018 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430357888}
I20240502 08:06:11.132787 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:11.118273 27421 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.134182 26646 raft_consensus.cc:2932] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:11.095445 26498 rpcz_store.cc:276] 0502 08:06:10.083697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.084467 (+1000770us) service_pool.cc:225] Handling call
0502 08:06:11.094799 (+ 10332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.152273 26496 rpcz_store.cc:276] 0502 08:06:09.863033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.976132 (+1113099us) service_pool.cc:225] Handling call
0502 08:06:11.113206 (+137074us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:11.153903 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 219181us
I20240502 08:06:11.166649 26576 raft_consensus.cc:2354] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:11.167327 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:11.184326 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 682) took 1297 ms. Trace:
I20240502 08:06:11.184986 26496 rpcz_store.cc:276] 0502 08:06:09.887037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.183264 (+1296227us) service_pool.cc:225] Handling call
0502 08:06:11.183662 (+ 398us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.183888 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.183927 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.183984 (+ 57us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.184041 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.184112 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.184253 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.187526 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 683) took 1284 ms. Trace:
I20240502 08:06:11.188417 26615 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.188151 26496 rpcz_store.cc:276] 0502 08:06:09.903324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.186517 (+1283193us) service_pool.cc:225] Handling call
0502 08:06:11.186913 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.187129 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.187164 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.187220 (+ 56us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.187260 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.187329 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.187462 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.154368 26542 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.068773 27425 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.190743 26542 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.191236 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 684) took 1269 ms. Trace:
I20240502 08:06:11.195026 26542 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.195986 26496 rpcz_store.cc:276] 0502 08:06:09.922190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.189799 (+1267609us) service_pool.cc:225] Handling call
0502 08:06:11.190613 (+ 814us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.190823 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.190862 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.190914 (+ 52us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.190957 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.191038 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.191171 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.198645 26542 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.199105 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 686) took 1213 ms. Trace:
I20240502 08:06:11.199561 26496 rpcz_store.cc:276] 0502 08:06:09.985508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.198255 (+1212747us) service_pool.cc:225] Handling call
0502 08:06:11.198576 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.198756 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.198790 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.198838 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.198879 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.198942 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.199049 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.201505 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 696) took 1209 ms. Trace:
I20240502 08:06:11.201910 26496 rpcz_store.cc:276] 0502 08:06:09.992233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.200645 (+1208412us) service_pool.cc:225] Handling call
0502 08:06:11.200948 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.201128 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.201186 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.201235 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:11.201275 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.201337 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.201451 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.160506 26541 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.136157 27439 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.212910 26542 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.203775 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 697) took 1197 ms. Trace:
I20240502 08:06:11.216567 26496 rpcz_store.cc:276] 0502 08:06:10.005870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.202960 (+1197090us) service_pool.cc:225] Handling call
0502 08:06:11.203264 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.203433 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.203455 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.203498 (+ 43us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.203537 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.203599 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.203723 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.152534 27369 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.144963 27466 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:11.224294 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:11.147073 27411 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.227175 26569 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.094620 26572 rpcz_store.cc:276] 0502 08:06:09.801621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.287943 (+486322us) service_pool.cc:225] Handling call
0502 08:06:10.988961 (+701018us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.092511 (+103550us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.093654 (+ 1143us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54001b9c50. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411bc59 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:11.093849 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1611433856}
I20240502 08:06:11.196985 27416 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.229493 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 679) took 1388 ms. Trace:
I20240502 08:06:11.195206 27419 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.237547 27406 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.240692 26498 rpcz_store.cc:276] 0502 08:06:09.841346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.197574 (+1356228us) service_pool.cc:225] Handling call
0502 08:06:11.228872 (+ 31298us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.229060 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.229095 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.229172 (+ 77us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:11.229216 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.229288 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.229422 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.241396 27406 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.241850 27406 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.154726 27343 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.243723 27343 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.244326 27343 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:11.229480 27466 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.211128 27323 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.247495 26494 raft_consensus.cc:2354] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:11.247856 27323 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.248345 27323 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.145766 26493 rpcz_store.cc:276] 0502 08:06:09.826861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.921060 (+1094199us) service_pool.cc:225] Handling call
0502 08:06:11.085603 (+164543us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:11.250229 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 310817us
I20240502 08:06:11.250900 26616 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.220036 27452 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.219835 27453 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.255298 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:11.219615 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:11.136430 26574 rpcz_store.cc:276] 0502 08:06:09.814080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:09.986809 (+172729us) service_pool.cc:225] Handling call
0502 08:06:10.987912 (+1001103us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54001b9c54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:10.988071 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1906184448}
I20240502 08:06:11.249114 26575 raft_consensus.cc:2379] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:11.196970 27436 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.273638 26569 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
I20240502 08:06:11.229271 27411 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.249014 26494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 705) took 1165 ms. Trace:
I20240502 08:06:11.240625 27416 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:11.337399 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 86564.2us
W20240502 08:06:11.331351 27470 log.cc:924] Time spent T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240502 08:06:11.342020 26616 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.251107 27419 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.344869 27419 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:11.344995 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 231004us
I20240502 08:06:11.345752 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.346973 27340 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.347283 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:11.348167 26495 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:11.348799 26495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 709) took 1104 ms. Trace:
I20240502 08:06:11.349239 26495 rpcz_store.cc:276] 0502 08:06:10.244019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.151469 (+907450us) service_pool.cc:225] Handling call
0502 08:06:11.347137 (+195668us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800005198. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba72e 0000000000339c39 00007f9708fe96c0 00007f97090af3d4 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:11.348731 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7835904}
I20240502 08:06:11.265866 27453 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.351138 26500 raft_consensus.cc:2354] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:11.332046 27452 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:11.354423 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 141582us
I20240502 08:06:11.355327 26467 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.357733 26467 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.343993 27416 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.352912 27453 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.360803 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.342372 27477 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.358449 26499 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:11.347333 27340 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.355345 26542 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.370874 26542 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.358552 27373 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:11.366503 26467 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.355901 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 685) took 1397 ms. Trace:
W20240502 08:06:11.365814 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 177504us
I20240502 08:06:11.360529 27452 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.340185 26494 rpcz_store.cc:276] 0502 08:06:10.083964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.014856 (+930892us) service_pool.cc:225] Handling call
0502 08:06:11.248796 (+233940us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54003a02d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:11.248949 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":531707264}
I20240502 08:06:11.361951 27438 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.365094 27145 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.462412 27438 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:11.462900 27438 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:11.462957 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 104723us
I20240502 08:06:11.396262 26645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 567) took 1146 ms. Trace:
I20240502 08:06:11.463783 27452 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.468820 26645 rpcz_store.cc:276] 0502 08:06:10.250301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.882520 (+632219us) service_pool.cc:225] Handling call
0502 08:06:10.882790 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:10.884354 (+ 1564us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:10.901443 (+ 17089us) log.cc:842] Serialized 14 byte log entry
0502 08:06:11.354097 (+452654us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.359397 (+ 5300us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:11.361192 (+ 1795us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.366175 (+ 4983us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.376727 (+ 10552us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3238,"prepare.queue_time_us":6563,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":10745,"raft.queue_time_us":173287,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":37834,"spinlock_wait_cycles":1042432,"thread_start_us":388212,"threads_started":2,"wal-append.queue_time_us":337020}
I20240502 08:06:11.470118 27434 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.471524 27409 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.472275 27434 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.473438 27435 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.465673 27413 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.412887 27342 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.413467 26615 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.368257 27426 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.366860 27477 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.399384 27410 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.373216 26542 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.427676 27418 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.429358 27283 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.432279 27333 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.433517 27421 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:11.435515 27419 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.354382 27428 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.403172 26467 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.467226 26650 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
I20240502 08:06:11.463055 27425 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.368860 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
W20240502 08:06:11.478822 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 65419.6us
W20240502 08:06:11.480013 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 138096us
I20240502 08:06:11.491130 26616 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:11.379065 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 218636us
I20240502 08:06:11.367784 27340 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.400135 27422 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.477632 27342 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.492000 26615 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:11.492606 26541 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.480237 27473 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:11.481153 27426 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.481329 27477 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.466363 27406 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:11.494282 26597 debug-util.cc:398] Leaking SignalData structure 0x7b08001cdc60 after lost signal to thread 25355
W20240502 08:06:11.512742 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 139620us
I20240502 08:06:11.489339 27409 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.488624 27410 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.514422 26541 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:11.481958 27405 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.492326 26616 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.554236 26492 raft_consensus.cc:2379] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:11.554924 26541 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.518023 26542 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.510892 27283 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.510105 27342 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:11.557665 26597 debug-util.cc:398] Leaking SignalData structure 0x7b080002afe0 after lost signal to thread 26433
W20240502 08:06:11.557792 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 154758us
I20240502 08:06:11.600116 26467 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.444173 26497 rpcz_store.cc:276] 0502 08:06:09.958928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:11.196792 (+1237864us) service_pool.cc:225] Handling call
0502 08:06:11.249620 (+ 52828us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:11.267233 (+ 17613us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:11.331663 (+ 64430us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:11.333569 (+ 1906us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:11.338551 (+ 4982us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:11.342816 (+ 4265us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:11.353984 (+ 11168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:11.602854 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.507158 27422 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.515159 27426 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.553347 27406 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:11.607988 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 53155.8us
I20240502 08:06:11.602912 27487 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.509524 27452 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.512127 27425 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:11.556802 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 49937.8us
W20240502 08:06:11.601318 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 109157us
I20240502 08:06:11.520864 27413 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.606544 27418 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.607913 27422 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.520826 27436 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.501600 27421 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.622023 27421 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.622514 27421 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.622686 27411 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:11.623773 27416 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.549854 27428 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.516191 27145 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.594573 27283 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.617179 27452 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.600067 27410 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:06:11.548354 27488 log.cc:924] Time spent T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240502 08:06:11.517592 27484 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.616575 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.615715 26615 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:11.608095 27420 consensus_meta.cc:287] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 0.535s user 0.005s sys 0.005s
I20240502 08:06:11.618932 27493 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:11.638844 27493 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.638814 27420 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.639844 27435 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:11.605130 27373 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.608042 27369 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.556660 27409 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.608389 27486 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.642191 27478 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.643389 27486 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.643491 27478 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.605695 27438 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:11.626792 26597 debug-util.cc:398] Leaking SignalData structure 0x7b08001052c0 after lost signal to thread 26525
I20240502 08:06:11.625146 27465 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.612538 27487 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.615269 27425 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.616079 27406 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.549469 27323 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.615528 27492 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.635056 27283 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.643985 27428 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.636425 27410 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.639365 27493 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:11.660396 27492 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.606935 27419 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.642076 27373 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.658663 27425 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.636008 27333 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.636775 27452 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.641101 27420 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.657119 27406 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.643936 27478 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:11.643847 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 141965us
I20240502 08:06:11.686623 27283 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.738497 26395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52756 (request call id 18) took 1765 ms. Trace:
I20240502 08:06:11.773025 27283 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.684973 27323 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:11.747658 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 229567us
I20240502 08:06:11.698630 27340 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:11.750504 26597 debug-util.cc:398] Leaking SignalData structure 0x7b0800332ce0 after lost signal to thread 26600
I20240502 08:06:11.774689 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:11.776371 27179 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 1 LEARNER]: Advancing to term 3
W20240502 08:06:11.777132 26597 debug-util.cc:398] Leaking SignalData structure 0x7b080032db00 after lost signal to thread 26674
I20240502 08:06:11.777009 26542 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.628127 26492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 289) took 1801 ms. Trace:
W20240502 08:06:11.777559 26597 debug-util.cc:398] Leaking SignalData structure 0x7b08002a8240 after lost signal to thread 26675
I20240502 08:06:11.778198 26492 rpcz_store.cc:276] 0502 08:06:09.827113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.991754 (+1164641us) service_pool.cc:225] Handling call
0502 08:06:11.627905 (+636151us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400367754. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:11.628064 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1170387712}
I20240502 08:06:11.779994 26542 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.757917 26562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a26db3828484577a5da525732f0b102 with delete_type TABLET_DATA_TOMBSTONED (TS 97dc9560dbbe4d1aa44b2b2edda4080f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51122
I20240502 08:06:11.638175 26616 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.751101 26644 raft_consensus.cc:1226] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240502 08:06:11.780357 26597 debug-util.cc:398] Leaking SignalData structure 0x7b08003269c0 after lost signal to thread 27204
I20240502 08:06:11.605387 26467 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.782891 27323 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.783679 27323 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.784555 27323 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:11.785235 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 179748us
I20240502 08:06:11.786530 26467 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:11.786880 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 148619us
I20240502 08:06:11.772097 26395 rpcz_store.cc:276] 0502 08:06:09.972792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:09.973051 (+ 259us) service_pool.cc:225] Handling call
0502 08:06:11.738095 (+1765044us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:10.754224 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:10.755040 (+ 816us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:10.755093 (+ 53us) write_op.cc:427] Acquired schema lock
0502 08:06:10.755119 (+ 26us) tablet.cc:660] Decoding operations
0502 08:06:10.755549 (+ 430us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:10.755653 (+ 104us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:10.757200 (+ 1547us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:06:10.758849 (+ 1649us) tablet.cc:699] Row locks acquired
0502 08:06:10.759070 (+ 221us) write_op.cc:260] PREPARE: finished
0502 08:06:10.760492 (+ 1422us) write_op.cc:270] Start()
0502 08:06:10.760611 (+ 119us) write_op.cc:276] Timestamp: P: 1714637170760255 usec, L: 0
0502 08:06:10.760634 (+ 23us) op_driver.cc:331] REPLICATION: starting
0502 08:06:10.761276 (+ 642us) log.cc:842] Serialized 532 byte log entry
0502 08:06:11.242100 (+480824us) op_driver.cc:444] REPLICATION: finished
0502 08:06:11.542496 (+300396us) write_op.cc:301] APPLY: starting
0502 08:06:11.550076 (+ 7580us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:11.550191 (+ 115us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:11.550336 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:11.550546 (+ 210us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:11.550705 (+ 159us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:06:11.550755 (+ 50us) write_op.cc:312] APPLY: finished
0502 08:06:11.550967 (+ 212us) log.cc:842] Serialized 22 byte log entry
0502 08:06:11.551125 (+ 158us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:11.551204 (+ 79us) write_op.cc:446] Released schema lock
0502 08:06:11.551247 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:11.551504 (+ 257us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:11.551532 (+ 28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":339853,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":267457,"apply.run_cpu_time_us":1914,"apply.run_wall_time_us":34144,"num_ops":1,"prepare.queue_time_us":21090,"prepare.run_cpu_time_us":47246,"prepare.run_wall_time_us":113293,"raft.queue_time_us":340508,"raft.run_cpu_time_us":13251,"raft.run_wall_time_us":162534,"replication_time_us":481336,"thread_start_us":528224,"threads_started":4,"wal-append.queue_time_us":121922}]]}
I20240502 08:06:11.787873 27333 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.788074 26616 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.788249 27283 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:11.789521 26597 debug-util.cc:398] Leaking SignalData structure 0x7b08003269a0 after lost signal to thread 27205
I20240502 08:06:11.789561 26616 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.795264 26616 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.797277 26615 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.777621 27340 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.698630 27342 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.680876 27369 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.664279 27373 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.805099 26467 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.682569 27406 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.643385 27409 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.698336 27410 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.642411 27411 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:11.689561 27416 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.798898 27420 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:11.695184 27421 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.811527 27409 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.812336 27179 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:11.697085 27425 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.698798 27438 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.688262 27439 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.677763 27452 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.818945 27420 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.689827 27453 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.819743 27420 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.822618 26501 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:11.822978 27439 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.690609 27465 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.629037 27466 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:11.824352 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 708) took 1580 ms. Trace:
I20240502 08:06:11.824941 26467 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.814023 27421 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.641472 27484 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.827811 27484 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.827925 27466 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.828013 27409 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.826465 27439 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.830056 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:11.660862 27492 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.831267 27409 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.692328 27493 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:11.833565 27428 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.687179 27477 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:11.834132 27428 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.643862 27486 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.835018 27428 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.835253 27499 tablet_replica.cc:330] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: stopping tablet replica
I20240502 08:06:11.699867 27426 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.703616 27422 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.839877 27426 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.841251 26467 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.706840 27478 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.841960 27426 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:11.842454 26571 raft_consensus.cc:2354] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e for earlier term 3. Current term is 4.
I20240502 08:06:11.842691 26467 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.843514 27426 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.844020 27426 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:11.693985 27435 raft_consensus.cc:686] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:11.844264 26467 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:11.844821 27145 leader_election.cc:398] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e for earlier term 3. Current term is 4.
I20240502 08:06:11.845345 27145 leader_election.cc:401] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240502 08:06:11.845420 27438 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.699354 27405 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.845958 27438 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.846436 27438 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.806699 27342 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.810487 27410 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.847263 27493 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:11.837306 27465 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.848277 27465 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.834630 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:11.816812 27425 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.658799 27487 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.824798 26501 rpcz_store.cc:276] 0502 08:06:10.243683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:10.959773 (+716090us) service_pool.cc:225] Handling call
0502 08:06:11.823999 (+864226us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b54000fd2d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:11.824217 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1929476352}
I20240502 08:06:11.849750 26492 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:11.814374 27473 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.838481 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:11.850634 27473 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.851101 27473 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.844070 27478 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.804504 27491 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.848639 27466 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:11.848685 27465 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.818230 27490 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.858229 27490 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.858675 27490 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.845961 26467 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.844681 27426 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.854460 26644 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.855557 27491 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.841852 27422 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.861501 27435 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:11.865694 27477 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.860692 27439 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.855063 27420 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.868837 26542 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.868177 27369 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.864801 27491 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.878990 27453 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.873296 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:11.872782 27406 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:11.885270 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 88335.6us
I20240502 08:06:11.885985 26616 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:11.904717 27406 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.906179 27484 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.909649 27484 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:11.917665 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 71839.4us
I20240502 08:06:11.917635 26615 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.886405 27420 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.907421 26395 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4), now has a pending config: VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1), VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2), VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3), VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4). New cstate: current_term: 2 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:11.927819 26644 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:11.928449 27477 raft_consensus.cc:2741] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240502 08:06:11.928436 27373 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.929919 27478 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.930485 27478 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.930989 27478 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.935628 27323 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.939997 27323 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.939949 27478 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.940438 27426 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.947633 27373 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.948470 26615 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:11.960515 27477 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.960665 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:11.953570 27499 raft_consensus.cc:2229] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:11.950532 27487 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.961975 27499 raft_consensus.cc:2258] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:11.960811 27478 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.963534 27487 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:11.948917 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 68921.2us
I20240502 08:06:11.960811 27426 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.961026 27477 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:11.956049 27486 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
W20240502 08:06:11.966903 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 112637us
I20240502 08:06:11.966354 26467 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:11.969336 27477 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.969931 27478 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:11.970650 27478 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.971355 27478 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:11.971307 27491 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.978818 27499 ts_tablet_manager.cc:1875] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:11.982834 26542 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.984421 27487 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:11.984877 27487 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:11.985033 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.985365 27487 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:11.985157 26542 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:11.988170 27428 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.989528 27473 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.991902 27490 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:11.994395 27426 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:11.967576 27438 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:11.994606 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 69645.1us
I20240502 08:06:12.002153 27428 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.003378 26467 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.005226 27490 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.005851 27473 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:11.993129 27492 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.009639 26616 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.010027 27466 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e for earlier term 3. Current term is 4.
I20240502 08:06:12.010160 27323 raft_consensus.cc:2726] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.010797 27487 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.008755 27486 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.015295 26542 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.018375 27438 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.020188 27426 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:12.020534 27473 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.029206 27473 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.029728 27473 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.030061 27487 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:11.981701 27477 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:12.032900 27477 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:12.020941 26616 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.034719 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.035169 26467 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.036101 27486 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.036887 27428 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.036958 27422 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:12.037302 27425 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.037492 27422 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.037689 27425 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.037918 27422 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.038130 27425 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.040722 27473 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.045100 27425 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.023655 27426 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:12.046164 27426 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.046926 27428 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.023487 27323 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.024952 26542 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.026721 27452 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.028604 27179 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:12.047300 27425 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:12.029796 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 61740.4us
I20240502 08:06:12.054005 27487 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.055416 27452 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.055538 26542 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.056444 27452 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.036773 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.058758 26467 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.059681 27342 consensus_queue.cc:1046] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:12.060067 26467 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.060921 27373 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.061177 27421 raft_consensus.cc:3032] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.028633 27478 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.061569 27373 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.061690 27421 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.062376 27373 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.063338 27478 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.062992 27421 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.065187 27421 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.068109 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.069037 27438 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:12.070744 27426 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.020977 27491 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.051084 27465 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.076442 27426 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.076858 27473 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.078478 27499 ts_tablet_manager.cc:1888] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:06:12.079059 27499 log.cc:1196] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root/wals/9a26db3828484577a5da525732f0b102
I20240502 08:06:12.081435 26386 catalog_manager.cc:4830] TS 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): tablet 9a26db3828484577a5da525732f0b102 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:12.050787 27492 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.056875 26542 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.083449 27452 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.085084 26542 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.071269 26467 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.047343 27428 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.086170 27452 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.086818 27465 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.086910 27452 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.074587 26616 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.087451 27452 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.086835 26542 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.088536 27487 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.089211 26542 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.089354 27465 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.088511 26616 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.089995 27487 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:12.090431 27487 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.090806 27487 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.091338 27373 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.057322 26615 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.093605 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.093837 27473 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.094363 27473 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:12.094801 27473 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.096480 27426 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.090819 26542 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.045181 27422 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.099054 27323 raft_consensus.cc:2726] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.099527 26542 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.099635 27422 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:12.100162 26542 raft_consensus.cc:2670] T 9a26db3828484577a5da525732f0b102 P 97dc9560dbbe4d1aa44b2b2edda4080f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:06:12.099753 27452 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.100903 26542 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.101696 27438 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.101990 26542 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.105515 27420 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.101758 27323 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.103545 27452 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.106082 27420 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.110617 27420 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.106117 27323 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:12.103884 27465 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.114748 27323 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.114786 27465 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:12.115476 27465 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.104635 26616 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.106011 26542 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.124433 27323 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.125651 26615 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.121619 27473 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.126914 27452 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.130574 26542 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.132552 27421 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.132697 26616 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.133055 27421 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:12.133347 27473 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.133549 27492 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.135376 27421 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.135896 26616 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.136651 27473 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.136992 27428 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.137563 27426 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.138163 27422 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:12.138610 27422 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.139048 27422 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.142596 27496 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.143055 27496 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.143419 27496 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.143410 27323 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.134879 27465 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.103075 27487 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.147473 27465 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.146162 26615 leader_election.cc:398] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.149331 26467 leader_election.cc:398] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.147645 27487 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.146142 27428 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.149377 26615 leader_election.cc:401] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.151330 26615 leader_election.cc:398] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.151121 27496 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.151705 26615 leader_election.cc:401] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.152446 27465 raft_consensus.cc:3037] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.153190 26542 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.154228 27487 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.157747 26542 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.159307 27496 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.159965 27477 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.160812 27420 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.161825 27496 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.163475 27420 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.162356 27421 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.164458 26616 leader_election.cc:398] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.165349 27421 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.167246 27421 consensus_queue.cc:1046] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.168179 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:12.168388 27421 raft_consensus.cc:3032] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.168731 27421 raft_consensus.cc:729] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.161562 26542 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.170331 26542 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.172369 27496 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.172534 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.172741 27465 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.149734 26467 leader_election.cc:401] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.173511 26644 raft_consensus.cc:2379] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.174897 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.175592 27426 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.176952 26467 leader_election.cc:398] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.177335 26467 leader_election.cc:401] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.177985 27486 raft_consensus.cc:3037] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.178776 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.181046 26644 raft_consensus.cc:2379] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.182446 26569 raft_consensus.cc:1226] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.184409 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:12.185009 26571 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:12.173086 27496 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:12.187930 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.189458 27438 consensus_queue.cc:1046] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.187937 27496 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:12.190403 27496 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.191826 27369 raft_consensus.cc:3032] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.192060 26571 raft_consensus.cc:1226] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.192214 27369 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.193152 27369 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.194250 27486 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.194552 27369 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.196195 26571 raft_consensus.cc:1226] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.198294 26571 raft_consensus.cc:1226] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.200066 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.204510 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.205687 26574 raft_consensus.cc:2379] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.209955 26467 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.210412 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.210726 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.211433 26646 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.212065 27486 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.212229 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.214219 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.214825 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.215727 26645 raft_consensus.cc:2379] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.215893 26644 raft_consensus.cc:2354] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.218268 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.218366 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.219053 26645 raft_consensus.cc:2379] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.218947 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.219391 27486 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.148072 27422 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.221325 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.223444 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.224983 26645 raft_consensus.cc:2379] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.225298 26649 raft_consensus.cc:1226] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.226620 26541 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.227414 27420 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.224102 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.223961 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.228516 26569 raft_consensus.cc:2379] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.228755 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.229313 26571 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:12.229636 26447 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 3ab970a1718349829f0f76fa212d5594
W20240502 08:06:12.229992 26447 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 3ab970a1718349829f0f76fa212d5594
I20240502 08:06:12.232721 27477 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.229640 26644 raft_consensus.cc:1226] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.240841 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.241814 26501 raft_consensus.cc:2454] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.242578 26569 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.243384 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.248129 26647 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:12.249231 26642 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.250804 26569 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:12.251755 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.252297 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.252799 26499 raft_consensus.cc:2379] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.253101 26494 raft_consensus.cc:2354] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.253484 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.253816 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.254319 26567 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.251911 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.252769 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.255312 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.255548 26571 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:12.258612 26501 raft_consensus.cc:2354] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.259229 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.248785 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.263466 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.264333 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.264416 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.265301 26648 raft_consensus.cc:2379] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.260011 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:12.266526 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:12.265825 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.265422 26575 raft_consensus.cc:2379] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.267966 26497 raft_consensus.cc:2396] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:12.261871 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.265344 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.269840 26648 raft_consensus.cc:1226] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.270289 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.270442 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.271508 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.241641 27421 consensus_queue.cc:260] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.273494 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.259006 26495 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.274883 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.275480 26498 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.276356 26643 raft_consensus.cc:1226] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.277135 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.278262 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.278599 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.279280 26497 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.280709 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.280903 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.282639 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.283319 26575 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.283524 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.283699 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.284307 26570 raft_consensus.cc:2379] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.284474 26573 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.285264 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.285964 26568 raft_consensus.cc:2379] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.286546 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.286943 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.287457 26572 raft_consensus.cc:2379] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.287705 26573 raft_consensus.cc:2379] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.288249 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.289067 26572 raft_consensus.cc:2354] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.286638 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.288352 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.269222 26494 raft_consensus.cc:1226] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.297827 26569 raft_consensus.cc:2379] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.297837 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.255213 27426 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.294512 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.255929 27496 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.295488 26568 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.299922 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.289193 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.302016 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.303534 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.305429 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.301115 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.305459 26647 raft_consensus.cc:1226] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.305827 26496 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.304162 26573 raft_consensus.cc:2354] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.304734 26649 raft_consensus.cc:1226] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.307130 26644 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.307583 26576 raft_consensus.cc:2379] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.308086 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.308382 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.308921 26576 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.309628 27421 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.310699 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.311293 26570 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.311726 26642 raft_consensus.cc:2354] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.308935 26572 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.316483 26571 raft_consensus.cc:2379] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.316823 26574 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.317951 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.318780 26568 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:12.319628 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.319628 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.320660 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.321275 26573 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.316618 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.313778 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.323380 26494 raft_consensus.cc:2379] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.324003 26499 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.325170 26615 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.320015 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.329144 27477 raft_consensus.cc:2781] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.329656 27477 raft_consensus.cc:686] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.329908 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.330792 26493 raft_consensus.cc:2354] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.330648 27477 consensus_queue.cc:237] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.333262 26570 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.333918 27422 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.337648 26574 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.338711 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.339341 26641 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.343559 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.344630 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.344816 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.345360 26643 raft_consensus.cc:2379] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.346016 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.346962 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.347695 26643 raft_consensus.cc:2379] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.348126 26569 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.348860 26498 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.346781 26649 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.349931 26500 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.301287 26567 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.351140 26642 raft_consensus.cc:2379] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.353996 26576 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.357880 26615 leader_election.cc:398] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.358914 26568 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.350803 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.361081 26647 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.362329 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.359795 26573 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.360241 26615 leader_election.cc:401] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.355819 26495 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.363991 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.363094 26501 raft_consensus.cc:2354] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.365846 26576 raft_consensus.cc:2354] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.366366 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.367208 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.367802 26571 raft_consensus.cc:2379] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.368831 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.316795 26646 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:12.370985 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.354401 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.374650 26616 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.354486 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.379652 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.380270 26615 leader_election.cc:398] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.381415 26615 leader_election.cc:401] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.382565 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.383074 26576 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.384217 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:12.386176 26572 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.387794 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.388617 26642 raft_consensus.cc:2379] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.388912 26647 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.389375 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.389977 26571 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.391301 26573 raft_consensus.cc:2379] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.392478 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.393422 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.394232 26567 raft_consensus.cc:2379] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.395045 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.395390 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.385936 26575 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:12.395759 26567 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.396016 26646 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.396463 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.397296 26575 raft_consensus.cc:2379] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.398079 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.396559 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.398936 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.402451 26575 raft_consensus.cc:2354] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.406652 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.406780 26576 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:12.407337 26568 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:12.407677 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.408236 26576 raft_consensus.cc:2354] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.411160 26498 raft_consensus.cc:2379] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.411273 26569 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:12.397326 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.411535 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.400991 26644 raft_consensus.cc:2354] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.417512 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.402032 26649 raft_consensus.cc:2379] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.415036 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.398417 27473 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.416070 26573 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.410732 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.418200 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.402681 27428 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.419857 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.416612 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.421898 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:12.424582 26571 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.417253 26645 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:12.425801 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.426304 26570 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.427064 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.427279 26570 raft_consensus.cc:1226] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.427811 26645 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.428776 26645 raft_consensus.cc:1226] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 5216c4a3a33f47ac88fa35704487f77e for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.424942 26576 raft_consensus.cc:1226] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.422269 26568 raft_consensus.cc:2379] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.422509 26575 raft_consensus.cc:2379] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
W20240502 08:06:12.419163 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 44622.8us
I20240502 08:06:12.423125 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.430900 26501 raft_consensus.cc:2354] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.432332 26573 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.432364 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.433574 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.434355 26498 raft_consensus.cc:2379] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.434427 26574 raft_consensus.cc:1226] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.433990 26643 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:12.429929 26567 raft_consensus.cc:2379] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:12.450590 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.437456 26575 raft_consensus.cc:1226] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.451872 26497 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:12.453059 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.453888 26494 raft_consensus.cc:2379] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.455210 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:12.436699 27496 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.460749 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.438318 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.462544 26645 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.439332 26573 raft_consensus.cc:1226] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.463514 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.439931 26641 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.440006 26646 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:12.440650 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.441738 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.442440 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.442523 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.442673 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:12.442744 26615 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.441772 26571 raft_consensus.cc:1226] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.449337 26572 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:12.464040 26494 raft_consensus.cc:1226] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 5216c4a3a33f47ac88fa35704487f77e for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.465802 27491 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.468539 26498 raft_consensus.cc:2379] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 2: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.468950 27466 consensus_queue.cc:1046] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:12.469530 26501 raft_consensus.cc:2396] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:12.471011 26496 raft_consensus.cc:2379] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.462455 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.438656 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.473024 26648 raft_consensus.cc:2379] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.437389 27428 consensus_queue.cc:1046] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:12.473779 26643 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:12.474553 27491 raft_consensus.cc:3032] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.475001 27491 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.465498 26616 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.476624 26568 raft_consensus.cc:1226] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.477931 26615 leader_election.cc:398] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.478320 26615 leader_election.cc:401] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.465914 26650 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:12.475895 27491 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.481860 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.482605 27491 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.483049 26646 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.483209 27465 consensus_queue.cc:1046] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:12.484382 27422 raft_consensus.cc:3032] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.484730 27422 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.478708 27428 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.485504 27422 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.480044 27473 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:12.486683 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.487494 26649 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.488339 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.488493 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.489068 26496 raft_consensus.cc:2379] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.480463 27492 consensus_queue.cc:1046] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:12.486845 27473 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.480294 26575 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.490737 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.494869 26498 raft_consensus.cc:2354] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.491250 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.491858 26542 leader_election.cc:398] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.495769 26542 leader_election.cc:401] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.497170 27422 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.499061 27342 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.500090 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.500823 26501 raft_consensus.cc:2379] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.494449 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.501201 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.496098 27473 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.498435 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.503373 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.503640 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.504186 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.504560 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.504752 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.505389 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.505831 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.505764 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.506610 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.503043 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:12.507158 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.507877 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.508567 26498 raft_consensus.cc:2379] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.504230 26497 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:12.510222 27496 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.510874 27342 raft_consensus.cc:3032] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.511270 27342 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.511670 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.512115 27342 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.513424 27342 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.508586 26500 raft_consensus.cc:2379] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.515620 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.515957 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.517249 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.520341 26497 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:12.501649 27421 consensus_queue.cc:1046] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.522475 26650 raft_consensus.cc:2379] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.528264 27473 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.529454 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:12.530385 26494 raft_consensus.cc:2354] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.531260 26647 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:12.522786 26616 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.525950 27466 raft_consensus.cc:3032] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.541729 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.542416 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.542910 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.543785 26494 raft_consensus.cc:2379] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.543691 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.544998 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.545717 26500 raft_consensus.cc:2354] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.546779 27369 consensus_queue.cc:1046] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.547410 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.547871 27369 raft_consensus.cc:3032] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.548264 26493 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.548295 27369 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.549907 27369 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.551251 27369 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.465423 27477 consensus_queue.cc:1046] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240502 08:06:12.569339 26542 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.570294 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:12.567354 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 53016us
I20240502 08:06:12.572786 27439 consensus_queue.cc:1046] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.570739 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:12.571205 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 48542.5us
I20240502 08:06:12.573415 26615 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.573890 27439 raft_consensus.cc:3032] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.574286 27439 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.574532 26616 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.575088 26615 leader_election.cc:398] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.575465 26615 leader_election.cc:401] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.575654 27428 raft_consensus.cc:3032] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.575187 27439 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.576030 27428 raft_consensus.cc:729] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.576112 26615 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.572821 26501 raft_consensus.cc:2379] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:12.576673 27439 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.577057 27421 raft_consensus.cc:2781] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.576929 27428 consensus_queue.cc:260] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.575649 26616 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.574318 27465 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.571926 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.571337 27422 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.534555 27466 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:12.579319 27425 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.577980 27428 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.582315 27496 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.581833 27473 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:12.587837 27422 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.595160 26615 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.593034 27340 consensus_queue.cc:1046] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240502 08:06:12.583602 27466 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:12.597394 27340 raft_consensus.cc:3032] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.597795 27340 raft_consensus.cc:729] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.594009 27477 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.590297 27421 raft_consensus.cc:686] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.590756 27496 raft_consensus.cc:686] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:12.601397 26542 leader_election.cc:398] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.602350 26542 leader_election.cc:401] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.586997 27438 consensus_queue.cc:1046] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240502 08:06:12.607241 27438 raft_consensus.cc:3032] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.607635 27438 raft_consensus.cc:729] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.601778 27340 consensus_queue.cc:260] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.608525 27438 consensus_queue.cc:260] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.609064 27340 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.609731 27438 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.586221 27426 consensus_queue.cc:1046] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240502 08:06:12.597868 27511 consensus_queue.cc:1046] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:12.618361 26615 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.599748 27422 raft_consensus.cc:686] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.602953 27425 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.611166 27477 raft_consensus.cc:686] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.619946 27490 raft_consensus.cc:3032] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.634820 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.622217 27487 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:12.637327 26569 raft_consensus.cc:1226] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 5216c4a3a33f47ac88fa35704487f77e for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.619177 27465 raft_consensus.cc:686] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
W20240502 08:06:12.638813 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 63233us
I20240502 08:06:12.625278 27420 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.641052 26542 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.649029 26616 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.627141 27515 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.635716 27490 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.638741 27486 raft_consensus.cc:3032] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.655149 26501 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.653584 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:12.638520 27425 raft_consensus.cc:686] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.659701 26496 raft_consensus.cc:1226] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.669039 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.657848 27486 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.661273 27420 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.678720 27425 consensus_queue.cc:237] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.680526 27439 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.680667 27369 consensus_queue.cc:1046] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:12.681116 27439 raft_consensus.cc:686] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.681816 27369 raft_consensus.cc:3032] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:12.682238 27369 raft_consensus.cc:729] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.669996 27422 consensus_queue.cc:237] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.656965 27421 consensus_queue.cc:237] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.683089 27369 consensus_queue.cc:260] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.684303 27369 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.672540 27516 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.672109 27515 raft_consensus.cc:686] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.686475 27486 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.656819 27496 consensus_queue.cc:237] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.653784 27487 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.689851 27487 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.690110 26497 raft_consensus.cc:1226] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.690248 26467 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.690521 27373 consensus_queue.cc:1046] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.691592 27373 consensus_queue.cc:1046] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:12.691887 26467 leader_election.cc:398] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.692273 26467 leader_election.cc:401] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.692525 27373 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.691803 26500 raft_consensus.cc:1226] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
W20240502 08:06:12.700028 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 66423.8us
I20240502 08:06:12.711257 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.711367 27527 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.712095 26467 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:12.712728 27527 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.713183 27527 raft_consensus.cc:686] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.714442 26541 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.715924 26541 leader_election.cc:398] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.716400 26576 raft_consensus.cc:1226] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.717317 26575 raft_consensus.cc:1226] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.716704 26541 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.719801 26541 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.721007 26568 raft_consensus.cc:1226] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.723919 26643 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.729048 26572 raft_consensus.cc:1226] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.731084 26575 raft_consensus.cc:1226] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.733511 26576 raft_consensus.cc:1226] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:12.734967 26572 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.735967 27487 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.737155 27487 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.738929 26571 raft_consensus.cc:1226] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:12.739612 27532 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.714087 27527 consensus_queue.cc:237] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.737851 27487 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.750911 26542 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.753533 27520 raft_consensus.cc:3032] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.754068 27520 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.757596 26467 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.696336 27420 raft_consensus.cc:686] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:12.758214 27520 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.759501 27520 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:12.760316 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 91417.8us
I20240502 08:06:12.761538 26616 leader_election.cc:398] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.762823 26616 leader_election.cc:398] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.763567 26616 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.755136 27521 raft_consensus.cc:3032] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
W20240502 08:06:12.764621 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 62247.8us
I20240502 08:06:12.765027 27521 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.759096 27420 consensus_queue.cc:237] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.764626 27514 raft_consensus.cc:2726] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.662667 27477 consensus_queue.cc:237] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.765837 27421 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.765925 27521 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.694023 27439 consensus_queue.cc:237] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.764653 26616 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:12.778905 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 77804.4us
I20240502 08:06:12.750752 27532 raft_consensus.cc:686] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.786641 26616 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.756570 27486 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.790652 27532 consensus_queue.cc:237] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.797717 26467 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.801726 26616 leader_election.cc:398] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.802868 26616 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.804425 26616 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.805775 26616 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.700428 27342 consensus_queue.cc:1046] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20240502 08:06:12.784301 27487 raft_consensus.cc:2781] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.687669 27490 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.816690 26542 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.817440 26616 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.819427 26542 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:12.821144 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.810526 27466 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.816051 27428 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.695240 27465 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:12.810573 27491 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.810985 27421 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.811594 27519 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.835736 26616 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.839637 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:12.815137 27473 consensus_queue.cc:1046] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240502 08:06:12.829622 27425 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.831646 27523 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:12.844468 26467 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.846769 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:12.834383 27532 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.835225 27535 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.847720 26643 raft_consensus.cc:2354] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.839076 27428 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.835877 27420 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.829849 27422 consensus_queue.cc:1046] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240502 08:06:12.841145 27517 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.834321 27373 raft_consensus.cc:686] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.837385 27487 raft_consensus.cc:686] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:12.858987 26467 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.859712 26542 leader_election.cc:398] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.784102 27515 consensus_queue.cc:237] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.854789 27478 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.834702 27492 consensus_queue.cc:1046] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240502 08:06:12.868139 26542 leader_election.cc:401] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:12.883677 26467 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:12.884279 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 48636.7us
W20240502 08:06:12.884522 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 141492us
I20240502 08:06:12.885056 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.888865 26541 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.864866 27539 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.866278 27519 raft_consensus.cc:686] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.866389 27530 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:12.867450 27425 raft_consensus.cc:686] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.897181 27478 raft_consensus.cc:686] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.895666 27530 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.900669 27478 consensus_queue.cc:237] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.895076 27426 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.902077 27530 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:12.902617 27530 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:12.902750 27425 consensus_queue.cc:237] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.903641 27487 consensus_queue.cc:237] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.905496 26616 leader_election.cc:398] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:12.907140 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.895325 27539 raft_consensus.cc:686] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.906483 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.911401 27421 raft_consensus.cc:3032] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:12.914548 27421 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:12.915654 27421 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.917122 27421 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:12.918943 26616 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.920092 26467 leader_election.cc:398] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:12.921041 26467 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.922358 26467 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.924402 26616 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:12.924700 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.923308 27342 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.895464 27523 raft_consensus.cc:686] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:12.894435 27519 consensus_queue.cc:237] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.906474 27492 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.908496 27496 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.897914 27373 consensus_queue.cc:237] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.928427 27340 raft_consensus.cc:2726] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:12.926367 27426 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.926213 27540 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:12.944267 27420 raft_consensus.cc:686] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:12.930619 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:12.945741 27426 raft_consensus.cc:686] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.932830 27539 consensus_queue.cc:237] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.937227 27522 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.927672 27438 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.962770 27420 consensus_queue.cc:237] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:12.968148 26616 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:12.969472 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 111836us
I20240502 08:06:12.970916 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:12.970901 27426 consensus_queue.cc:237] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:12.982784 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 93988.6us
W20240502 08:06:12.991951 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 61513us
I20240502 08:06:12.992798 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:12.950556 27529 log.cc:924] Time spent T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20240502 08:06:12.946202 27536 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.994601 26541 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.974375 27521 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:12.995537 27540 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:12.993625 27439 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:12.983455 27465 consensus_queue.cc:1046] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240502 08:06:13.006640 27439 raft_consensus.cc:686] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.001139 27340 consensus_queue.cc:1046] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.994477 27522 raft_consensus.cc:686] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:12.994019 27438 raft_consensus.cc:686] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.007323 26541 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.000301 27369 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.007547 26493 raft_consensus.cc:1226] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.013217 27541 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.010857 27439 consensus_queue.cc:237] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.014076 27532 consensus_queue.cc:1046] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:13.009697 27540 raft_consensus.cc:686] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.009193 27536 raft_consensus.cc:686] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.014168 27426 consensus_queue.cc:1046] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:12.973670 27523 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:13.023121 27545 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.026757 27540 consensus_queue.cc:237] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.028487 27543 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.028638 27490 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.029253 26541 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:12.997040 27516 consensus_queue.cc:1046] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240502 08:06:13.002184 27492 consensus_queue.cc:1046] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240502 08:06:13.032522 27518 raft_consensus.cc:3032] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.013582 27438 consensus_queue.cc:237] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.048137 27518 raft_consensus.cc:729] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.050729 27518 consensus_queue.cc:260] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.051946 27518 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.054641 27535 consensus_queue.cc:1046] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:13.057811 27533 consensus_queue.cc:1046] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240502 08:06:13.063218 27533 raft_consensus.cc:3032] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.063733 27533 raft_consensus.cc:729] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.071832 26650 raft_consensus.cc:1226] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.109155 26576 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.112026 26648 raft_consensus.cc:1226] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.113308 26647 raft_consensus.cc:1226] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.115990 26569 raft_consensus.cc:1226] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
W20240502 08:06:13.116811 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 87641.6us
I20240502 08:06:13.063773 27522 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:13.117980 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 65939.9us
I20240502 08:06:13.118997 26643 raft_consensus.cc:1226] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.119256 26571 raft_consensus.cc:1226] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.120757 26541 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.117801 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:13.123669 27552 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20240502 08:06:13.118829 27545 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.106393 27532 consensus_queue.cc:1046] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240502 08:06:13.137825 27426 consensus_queue.cc:1046] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.138415 26541 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.138563 27532 raft_consensus.cc:3032] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.105345 27533 consensus_queue.cc:260] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.139305 26572 raft_consensus.cc:1226] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.123684 27536 consensus_queue.cc:237] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.174948 27340 raft_consensus.cc:3032] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:13.175554 27340 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.178706 26541 leader_election.cc:398] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:13.139171 27532 raft_consensus.cc:729] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.187248 26541 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:13.194658 27549 log.cc:924] Time spent T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20240502 08:06:13.176628 27340 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.196627 27340 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:13.198333 27544 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 08:06:13.203576 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 75239.9us
I20240502 08:06:13.205730 27545 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.168634 27543 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.209200 26541 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.216562 26494 raft_consensus.cc:1261] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.223047 27542 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240502 08:06:13.224951 27543 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.226347 27532 consensus_queue.cc:260] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.226866 26494 raft_consensus.cc:1261] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.227569 27532 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.207156 26649 raft_consensus.cc:1226] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.232637 27531 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:13.233124 27531 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.228623 27421 consensus_queue.cc:1035] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240502 08:06:13.234565 27531 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.244693 27514 raft_consensus.cc:3032] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.245283 27514 raft_consensus.cc:729] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.246444 27514 consensus_queue.cc:260] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.259727 27534 raft_consensus.cc:3032] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:13.260304 27534 raft_consensus.cc:729] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
W20240502 08:06:13.272617 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 63507.4us
I20240502 08:06:13.273685 26541 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.282308 26500 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.285599 26646 raft_consensus.cc:1226] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:13.290620 27491 raft_consensus.cc:3032] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.291155 27491 raft_consensus.cc:729] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.292198 27477 raft_consensus.cc:3032] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.292657 27477 raft_consensus.cc:729] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.293666 27477 consensus_queue.cc:260] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.295064 27477 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.238641 27369 raft_consensus.cc:3032] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.298359 27369 raft_consensus.cc:729] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.306596 26649 raft_consensus.cc:1261] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.309015 27422 raft_consensus.cc:3032] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.309537 27422 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:13.261327 27534 consensus_queue.cc:260] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.314764 27533 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.314996 27534 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:13.317216 27422 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.318979 27422 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.330276 27514 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.332841 26500 raft_consensus.cc:1261] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.337132 26497 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.341243 26646 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.344681 27491 consensus_queue.cc:260] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.346571 27491 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:06:13.356958 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 83352.9us
I20240502 08:06:13.357815 27544 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240502 08:06:13.359123 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 50591.8us
I20240502 08:06:13.359759 27545 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.392884 27545 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.369848 27425 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.400008 26541 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.401098 27544 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.279511 26395 catalog_manager.cc:5484] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:13.375509 27179 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:13.383666 27542 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:13.385377 27473 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240502 08:06:13.388919 27369 consensus_queue.cc:260] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.411770 27369 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:13.405701 26541 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.418793 27544 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.420416 26541 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.422495 26541 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.424002 26541 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.421397 27544 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.436466 26541 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.427702 27545 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.454689 27530 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.470331 27477 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:13.498842 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 73321.4us
I20240502 08:06:13.478761 26641 raft_consensus.cc:1261] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.511404 27530 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 08:06:13.511970 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 75613.3us
I20240502 08:06:13.512910 26541 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.527366 26541 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.533524 27544 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.501083 27490 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240502 08:06:13.535312 26541 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.541710 27530 consensus_queue.cc:1035] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.510848 27477 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:13.556349 26541 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.554484 27487 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.556116 27530 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:13.572551 27487 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.574018 26541 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.578850 27487 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.580528 26541 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.581697 27487 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.584546 26541 leader_election.cc:398] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:13.587694 26541 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.600463 27477 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:13.622992 26541 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.630409 26541 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:13.614639 27420 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.041s sys 0.044s
W20240502 08:06:13.636683 26467 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.015s sys 0.020s
I20240502 08:06:13.655150 26500 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.636665 27530 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.635713 27496 raft_consensus.cc:2726] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.635649 27545 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:13.635625 27544 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240502 08:06:13.657536 26467 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.521s user 0.016s sys 0.020s
W20240502 08:06:13.654448 27420 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.041s sys 0.044s
W20240502 08:06:13.661782 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 527907us
W20240502 08:06:13.664065 27439 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.004s sys 0.033s
W20240502 08:06:13.670542 27439 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s user 0.004s sys 0.034s
I20240502 08:06:13.681881 26496 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.683686 26541 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:13.689384 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 49250.9us
I20240502 08:06:13.690155 26541 leader_election.cc:398] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Vote denied by peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577) with higher term. Message: Invalid argument: T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3.
I20240502 08:06:13.692482 26541 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.701210 26576 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240502 08:06:13.681772 27536 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.017s sys 0.013s
I20240502 08:06:13.733531 26568 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.677299 27520 consensus_queue.cc:1046] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240502 08:06:13.678841 27373 consensus_queue.cc:1046] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20240502 08:06:13.735594 26569 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.736105 26571 raft_consensus.cc:1226] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.675932 27532 consensus_queue.cc:1046] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240502 08:06:13.738814 26499 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.739852 27369 raft_consensus.cc:3032] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:13.740206 27369 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.740949 26499 raft_consensus.cc:1261] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.741536 27520 raft_consensus.cc:3032] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:13.741928 27520 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.742763 27520 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.744098 27520 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:13.701133 27545 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:13.754221 27496 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:13.754360 27545 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.754838 27496 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.756481 27545 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:13.757185 27496 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:13.702178 27544 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240502 08:06:13.758447 27558 log.cc:924] Time spent T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20240502 08:06:13.761536 26541 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.761317 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:13.763352 26541 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:13.735555 27536 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.017s sys 0.014s
I20240502 08:06:13.724679 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:13.770884 27491 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:13.771489 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:13.771210 27514 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:13.770773 27542 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240502 08:06:13.780530 27521 raft_consensus.cc:3032] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:13.781770 27521 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:13.764360 27420 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.784735 26571 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.793599 26571 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.794209 26573 raft_consensus.cc:1226] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.794932 26495 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.795753 26572 raft_consensus.cc:1261] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.808940 26571 raft_consensus.cc:1226] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:13.812175 27531 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:13.796571 27545 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:13.814561 27536 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:13.814481 27545 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.815166 27536 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.816946 27545 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:13.817077 27536 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:13.777027 27496 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.829813 26573 raft_consensus.cc:1261] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.778543 27544 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:13.829332 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:13.782724 27521 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.740993 27369 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:13.861358 27564 log.cc:924] Time spent T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20240502 08:06:13.847546 27527 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:13.834296 27491 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20240502 08:06:13.862954 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:13.886000 27542 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:13.903095 26572 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20240502 08:06:13.903755 26572 pending_rounds.cc:76] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Aborting all ops after (but not including) 2
I20240502 08:06:13.915158 26568 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.917779 26576 raft_consensus.cc:1261] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.908069 26572 pending_rounds.cc:97] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20240502 08:06:13.920339 26572 raft_consensus.cc:2894] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20240502 08:06:13.920877 26572 raft_consensus.cc:2866] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:13.875242 27527 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:13.969122 26647 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.969813 26569 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.968043 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:13.971275 26576 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:13.962699 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:13.975499 26574 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.975827 26647 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.984616 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:13.985375 26645 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:13.985813 26567 raft_consensus.cc:1261] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.920596 27439 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20240502 08:06:13.986682 26570 raft_consensus.cc:1261] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.988466 26650 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.991276 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:13.992141 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:13.901144 27567 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20240502 08:06:13.996464 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:13.996814 26574 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.995285 26615 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:13.900282 27542 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.001564 26571 raft_consensus.cc:1261] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:13.900398 27572 consensus_queue.cc:1046] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.098s
W20240502 08:06:13.988176 27554 log.cc:924] Time spent T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.076s user 0.000s sys 0.005s
I20240502 08:06:14.006947 26574 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 08:06:14.007792 27577 log.cc:924] Time spent T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20240502 08:06:14.009513 27477 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:14.022661 26492 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:14.023232 26574 raft_consensus.cc:1261] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.023468 26649 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:13.912186 27565 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20240502 08:06:13.980444 27539 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:14.027052 27539 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.028560 26498 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:14.029670 27539 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:13.961163 27438 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20240502 08:06:14.034274 27569 log.cc:924] Time spent T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
I20240502 08:06:14.035375 27491 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:14.038373 27534 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:14.014894 27439 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20240502 08:06:14.043061 26646 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:14.048295 27527 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.049923 27542 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:14.062206 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 82051.8us
I20240502 08:06:14.079147 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.080209 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.086298 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.088891 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.100021 26575 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:14.101680 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.102367 26467 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.102632 26501 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.104969 26615 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:14.111032 27527 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:14.113135 27527 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:14.114233 27520 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.114742 27520 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.115150 27520 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.118345 27536 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:14.120545 27536 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:14.125478 27567 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:14.059132 27542 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:14.100369 27561 log.cc:924] Time spent T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20240502 08:06:14.067133 27496 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.131104 26467 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:14.143936 26467 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:14.154135 26570 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:14.132077 27576 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.177655 27593 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 08:06:14.177963 27583 log.cc:924] Time spent T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20240502 08:06:14.179760 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.180686 26575 raft_consensus.cc:2379] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:14.146693 27486 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:14.178298 27593 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.186481 27593 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:14.147204 27527 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:14.130600 27567 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240502 08:06:14.130451 27426 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240502 08:06:14.130000 27520 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.195252 27490 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.144191 27542 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.147347 27496 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.196986 26646 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.200539 27527 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.207383 27531 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.216338 27531 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.198184 27490 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.212116 27491 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.179476 27576 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:14.258622 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.239045 27486 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.291887 27426 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.291538 27486 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.292436 27426 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.295641 27596 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.218618 27527 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.302774 27596 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:14.293962 27486 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.294414 27426 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.295939 26499 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.233935 27491 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:14.308748 27557 log.cc:924] Time spent T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240502 08:06:14.317787 27596 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:14.309108 27520 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.336566 27596 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.352669 27596 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.329619 27496 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.368288 27531 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.369544 26494 raft_consensus.cc:1261] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.378623 27490 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.379338 27490 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.379870 27490 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.382776 27542 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.384127 27596 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.377174 27593 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.398286 27527 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.400240 27527 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.400878 27527 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.404145 27527 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.410694 26647 raft_consensus.cc:1261] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.412400 27576 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.412935 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.413672 26495 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:14.415272 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.416087 26647 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:14.417253 27145 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:14.419085 27593 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.419939 27490 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.424643 27576 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.425143 27576 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.425483 27576 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.426445 27521 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:14.434023 27369 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:14.443885 27491 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.447901 27545 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.448377 27565 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.448483 27545 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.449174 27565 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.450651 27544 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.451249 27544 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.453552 27544 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.455024 27545 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.455293 27539 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.455941 27539 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.460350 27565 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.459925 26399 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 1 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:14.462550 27539 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.488175 27491 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.488801 27491 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.490875 27491 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.491909 27602 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.492385 27602 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.493849 27490 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.494505 27490 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.495185 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.495800 26572 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:14.496376 27490 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.496764 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.497722 26572 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.506413 26467 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.509254 27536 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:14.509747 27536 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.510203 27536 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:14.510069 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.499397 27602 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.522336 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:14.521677 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 416794us
I20240502 08:06:14.523463 27565 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.523833 27565 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.524184 27565 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.514766 26572 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.528599 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.529383 26572 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:14.530706 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.532616 26572 raft_consensus.cc:2379] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:14.533150 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.535236 26495 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:14.536693 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.536835 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.536720 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:14.537564 26495 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.537868 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.538568 27340 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:14.538586 27539 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.539927 26647 raft_consensus.cc:1261] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:14.540174 27539 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.540535 27539 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.542732 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.543458 26495 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.543658 27612 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.544261 27612 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.546649 27612 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:14.548656 27573 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.025s sys 0.093s
I20240502 08:06:14.549070 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:14.551635 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 359301us
I20240502 08:06:14.553201 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.553938 26646 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:14.551873 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.559816 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:14.563144 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.563901 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.564661 26567 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:14.565347 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.565958 26498 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:14.567018 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.567765 26575 raft_consensus.cc:2379] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:14.568513 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:14.569057 26575 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.570358 27340 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:14.570912 27340 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.571233 27439 consensus_queue.cc:1035] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:14.571259 27340 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:14.573885 26541 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:14.574012 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.576237 26570 raft_consensus.cc:2379] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:14.577267 27605 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:14.549070 27573 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.026s sys 0.093s
I20240502 08:06:14.587914 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.610467 27439 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.588807 27607 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.610985 27439 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.611388 27607 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.590027 27422 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.590276 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.590348 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.617319 26493 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.532143 27536 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.594619 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:14.624255 27425 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.596074 26641 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:14.601022 27514 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.630913 27514 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.634533 27620 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:14.602486 26542 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.604492 27609 raft_consensus.cc:1050] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
W20240502 08:06:14.640714 27609 raft_consensus.cc:1054] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Unable to promote non-voter 5216c4a3a33f47ac88fa35704487f77e: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:06:14.640781 27605 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.641434 27605 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.641780 27605 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:14.646271 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 63900.1us
I20240502 08:06:14.644610 27496 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.648160 27496 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.648911 27425 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:14.646479 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 72673.3us
I20240502 08:06:14.651638 26541 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:14.659847 27523 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:14.656239 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.590881 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.672482 26541 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:14.679270 27514 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.613289 27422 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.682802 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.672988 26575 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:14.689033 26642 raft_consensus.cc:1261] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:14.689451 26649 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.691417 26615 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:14.693436 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.694139 27530 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:14.667929 27536 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.701603 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.702144 26616 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:14.696671 26467 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:14.648591 27607 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.695602 26642 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:14.683439 26495 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.694396 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.713986 26541 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:14.714550 27603 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240502 08:06:14.671165 27523 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20240502 08:06:14.712314 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:14.702931 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:14.731403 27610 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:14.733342 27565 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.727397 26575 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:14.707226 27530 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.710176 27426 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:14.687100 26644 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.710582 27490 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:14.718652 27496 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:14.784266 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 55977.3us
I20240502 08:06:14.728642 27542 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.715590 27567 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20240502 08:06:14.744369 26648 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:14.742693 27439 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.775658 27514 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:14.730875 26649 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.728977 27607 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20240502 08:06:14.728480 27477 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20240502 08:06:14.728777 27596 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240502 08:06:14.691679 27609 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20240502 08:06:14.745081 27530 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.735404 27487 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20240502 08:06:14.777092 27420 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
W20240502 08:06:14.791805 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 46801.1us
I20240502 08:06:14.829576 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.848855 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.865768 26572 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:14.860749 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.838943 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.832746 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.866247 26642 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:14.868981 26568 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.868338 26650 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.848062 27426 raft_consensus.cc:686] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:14.879014 27534 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.851617 27539 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.879547 27534 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.880918 27539 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.881485 27534 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.880666 27426 consensus_queue.cc:237] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.882486 27422 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.882625 27486 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.882762 27533 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.883180 27486 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.931702 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.883352 27533 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:14.932324 27530 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.932854 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:14.933622 26572 raft_consensus.cc:2379] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:14.931334 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.881444 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.792670 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.786484 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:14.937170 26571 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:14.824149 26575 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
W20240502 08:06:14.852821 27629 log.cc:924] Time spent T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.173s user 0.000s sys 0.002s
W20240502 08:06:14.943869 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 79064.9us
I20240502 08:06:14.857977 27425 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:14.936522 27565 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.853513 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.871726 27514 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:14.790553 27610 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.777392 27576 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:14.943372 27572 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:14.788666 27491 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.167s
I20240502 08:06:14.982219 27594 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.960116 27530 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:14.962677 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 244450us
I20240502 08:06:14.980379 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:14.969918 27533 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:14.980588 26567 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:14.985404 27602 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:14.969357 27588 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:14.995286 27600 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.015772 26467 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:14.790885 27542 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.063201 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:15.000921 27514 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.069121 27592 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.069720 27592 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.070174 27533 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:15.070694 27533 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.071112 27533 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.022460 27596 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.015571 27425 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:15.059582 27614 log.cc:924] Time spent T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240502 08:06:15.059427 27594 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.046418 27632 log.cc:924] Time spent T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20240502 08:06:15.021855 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:15.103724 27635 log.cc:924] Time spent T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20240502 08:06:15.104100 27600 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.072417 27612 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.022150 27340 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.089546 27596 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.045768 27565 consensus_queue.cc:1035] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20240502 08:06:15.086902 27373 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.065088 27602 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.263684 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 247989us
I20240502 08:06:15.265317 26467 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.072851 27530 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.128639 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 401368us
I20240502 08:06:15.269295 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.262157 27596 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.200634 27594 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.262976 26495 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.201658 27534 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.276856 26615 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:06:15.266283 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 220730us
I20240502 08:06:15.272653 27473 consensus_queue.cc:1035] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:15.277276 27592 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.279690 27420 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.271449 27542 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.021940 27572 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.280725 27420 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.281857 27473 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:15.284132 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:15.284802 26574 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.274557 27486 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.084596 27588 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.276043 26650 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.268898 27373 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.282757 27572 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.291388 27420 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.292385 26542 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:15.294376 27613 log.cc:924] Time spent T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240502 08:06:15.294391 27596 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:15.299751 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.307142 26615 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.346202 26467 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.293933 27576 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.345348 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.351190 26650 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.314486 26615 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.276893 27534 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.296818 27628 log.cc:924] Time spent T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240502 08:06:15.356519 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:15.310571 27592 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:15.355515 27616 log.cc:924] Time spent T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240502 08:06:15.355820 27527 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:15.297104 27491 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20240502 08:06:15.295079 27596 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:15.145820 27612 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.347014 27638 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:15.118429 27425 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.047616 27610 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.360131 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.397187 26650 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:15.312472 26616 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:15.201048 27340 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.375198 26646 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:15.362111 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 47740.5us
I20240502 08:06:15.362922 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:15.401526 26647 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:15.349812 26648 consensus_meta.cc:287] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.512s user 0.008s sys 0.001s
I20240502 08:06:15.376112 27527 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.129602 27605 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.370183 27611 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.310775 27420 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.296207 27542 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
W20240502 08:06:15.382572 26645 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.013s sys 0.026s
I20240502 08:06:15.386793 27609 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:15.353753 27544 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.310083 27620 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:15.296967 27422 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
W20240502 08:06:15.384371 27627 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.021s sys 0.033s
I20240502 08:06:15.368978 27477 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:15.369587 27592 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:15.361258 26574 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
W20240502 08:06:15.399616 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 89786.5us
I20240502 08:06:15.399529 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:15.437352 26650 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.404103 27652 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.342687 27486 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:15.391942 27596 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.407444 26615 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:15.388499 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.415316 27610 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.403476 27439 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.406458 27340 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.409322 27527 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.407140 27473 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:15.406973 26648 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:15.441780 26616 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:15.421895 26645 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.013s sys 0.026s
I20240502 08:06:15.447211 26647 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:15.463480 26650 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:15.464598 27572 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.465404 27520 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.466163 27522 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.494323 27520 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.494469 27522 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.442798 27592 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.485419 27620 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240502 08:06:15.473626 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 66289.5us
I20240502 08:06:15.504432 26615 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.474431 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.475636 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.422614 27420 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.434103 27609 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.447834 27486 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.474144 27530 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:15.470528 27426 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.015s sys 0.047s
I20240502 08:06:15.490648 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.490423 27572 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.491191 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.491335 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.491420 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.476888 27340 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.497849 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:15.447286 27627 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.789s user 0.021s sys 0.035s
I20240502 08:06:15.501439 27576 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:15.473088 26644 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.467012 27369 raft_consensus.cc:482] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.516359 27542 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:15.472666 27610 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.482405 27373 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.479287 27473 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:15.519039 26650 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:15.521951 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.442636 27611 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.526185 26647 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 1.
I20240502 08:06:15.552129 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:15.517685 27639 log.cc:924] Time spent T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240502 08:06:15.503219 27602 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.503765 27620 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:06:15.525321 27426 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.016s sys 0.047s
I20240502 08:06:15.433605 27544 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.520846 27486 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:15.577244 26574 raft_consensus.cc:1226] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:15.479804 27439 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.583319 26496 raft_consensus.cc:1261] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:15.583999 27612 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.549557 27534 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.554543 27600 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.595798 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.596284 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.556036 27657 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.606199 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:15.609853 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:15.614010 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.572292 27588 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.575217 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.625375 27654 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.625651 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.586140 26493 raft_consensus.cc:1261] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:15.586724 26498 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:15.586856 27653 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.586565 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.454900 27652 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.579031 27369 raft_consensus.cc:504] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.588501 27610 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.580299 27576 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:15.750404 26650 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.699227 27636 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:15.751219 27636 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.734575 27532 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.675362 27611 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.730633 27539 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:15.728336 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 286628us
I20240502 08:06:15.700624 27605 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:15.675853 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 127676us
I20240502 08:06:15.594139 27473 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.755340 26500 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.754680 26572 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:15.764005 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.767769 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.757045 26569 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.755860 26576 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:15.646420 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:15.769764 26568 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:15.748627 26497 raft_consensus.cc:2396] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:15.604271 27638 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:15.763387 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.775363 26498 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.751475 26574 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:15.768981 26642 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:15.770412 26650 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
W20240502 08:06:15.748447 27630 log.cc:924] Time spent T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240502 08:06:15.605738 27425 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.791153 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.624541 27620 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.795601 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.753373 27636 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.804734 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.809679 26500 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:15.809904 26571 raft_consensus.cc:2379] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:15.758266 27611 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.752249 27532 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.747378 27609 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:15.728741 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:15.835532 27648 log.cc:924] Time spent T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20240502 08:06:15.836479 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.687561 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:15.842006 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.844125 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:15.845788 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 341428us
I20240502 08:06:15.753353 27600 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.772115 27439 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.762205 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.855608 26467 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:15.857568 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.753069 27522 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.885107 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:15.865924 26616 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:15.872021 26499 raft_consensus.cc:2396] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:15.779310 26648 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:15.800763 27612 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.828506 27638 raft_consensus.cc:686] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:15.791628 27654 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.771966 27539 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.872649 27514 consensus_meta.cc:287] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.803s user 0.011s sys 0.000s
I20240502 08:06:15.858572 27420 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.857319 27636 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.888127 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.888520 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.899811 26571 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:15.900820 27659 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:15.877872 26567 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.901378 27659 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:15.902590 26616 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:15.887651 27439 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:15.889966 27542 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:15.878350 27600 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.862349 26568 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:15.887181 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:15.873099 27544 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.850708 27611 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.901037 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.762835 27653 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.919483 26571 raft_consensus.cc:2379] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 2: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:15.917953 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.890637 27596 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.907255 27654 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.901876 27659 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:15.876048 26644 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:15.765651 27657 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.905213 27514 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.916090 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.969548 26572 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:15.916124 26492 raft_consensus.cc:2396] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:15.900122 26542 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:15.920464 26568 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:15.897783 27340 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.955051 26569 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:15.963989 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.964604 26647 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:15.966248 26541 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:15.951463 26493 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:15.977521 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.976693 26575 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:15.978353 26647 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:15.979408 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.980072 26647 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:15.980844 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.970932 26574 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:15.968664 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:15.981820 26571 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:15.919198 27439 raft_consensus.cc:686] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:15.897569 27612 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:15.965720 26576 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:15.987840 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:15.911635 27636 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:15.989953 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 89900.2us
I20240502 08:06:15.993381 26542 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:15.994689 27420 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:15.997805 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:16.004575 26571 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240502 08:06:15.988242 27650 log.cc:924] Time spent T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.299s user 0.000s sys 0.001s
I20240502 08:06:16.005424 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:15.981010 27340 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:15.992628 26646 consensus_meta.cc:287] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.580s user 0.012s sys 0.000s
W20240502 08:06:16.019696 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 53590.1us
I20240502 08:06:16.018896 27652 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.985347 27373 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:15.991120 27588 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:15.988581 27369 leader_election.cc:288] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.022002 26541 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:16.186280 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 164363us
I20240502 08:06:15.987916 27610 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:15.995128 27420 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:16.023252 26542 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:16.011792 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.025163 26467 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:16.193264 26571 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:16.082808 27373 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.027985 27340 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.088958 27522 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:16.023692 26646 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:16.020496 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.075511 27532 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.087834 26494 raft_consensus.cc:2396] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:16.258499 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.258767 27340 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.259099 27145 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:16.156944 27663 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:16.091781 27592 consensus_meta.cc:287] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 0.593s user 0.007s sys 0.014s
W20240502 08:06:16.082553 27533 consensus_meta.cc:287] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 1.011s user 0.018s sys 0.005s
I20240502 08:06:16.088666 27534 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.093772 27671 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:16.094560 27642 log.cc:924] Time spent T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240502 08:06:16.004878 27662 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:16.078336 27527 consensus_meta.cc:287] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.591s user 0.009s sys 0.000s
I20240502 08:06:16.124337 27576 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.156505 27611 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.027079 27496 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:16.170154 26495 consensus_meta.cc:287] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.859s user 0.010s sys 0.002s
I20240502 08:06:16.282639 27698 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.282824 26495 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:16.189049 26541 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:16.285135 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 96202.6us
I20240502 08:06:16.266798 27612 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.177769 26500 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:16.195302 27669 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:16.170997 27657 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.095949 27439 consensus_queue.cc:237] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.170017 27652 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.211773 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.350219 26571 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:16.174686 27600 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.217777 27655 consensus_queue.cc:1035] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240502 08:06:16.177877 27653 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.221796 27373 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.222245 27682 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.227239 27687 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.236812 27691 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:16.239760 27697 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.241236 27666 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.242566 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.186507 27596 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:16.257699 26644 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:16.259240 26646 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:15.991144 27638 consensus_queue.cc:237] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.197554 27420 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:16.198325 27369 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:16.194725 27514 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:16.191722 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.213609 27610 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:16.270313 27674 log.cc:924] Time spent T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
W20240502 08:06:16.355161 26495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 10) took 1286 ms (client timeout 1577 ms). Trace:
I20240502 08:06:16.274235 27699 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.189860 27588 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.265533 27532 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.262073 27522 raft_consensus.cc:686] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:16.266575 27663 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:16.281622 27576 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:16.282091 27611 raft_consensus.cc:482] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.263025 27667 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.373994 26567 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
W20240502 08:06:16.277413 27670 log.cc:924] Time spent T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20240502 08:06:16.286640 26541 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:16.168179 27539 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.268683 27636 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:16.354791 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.455432 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.282649 27496 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.264564 27592 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.267525 27533 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:16.362455 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 339246us
I20240502 08:06:16.466032 26542 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:16.367043 26568 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:16.371217 27596 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.279512 27527 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.355974 27612 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:16.282644 27473 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.474949 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.446100 27532 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.476317 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.476969 27473 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:16.369098 27486 consensus_meta.cc:287] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.742s user 0.011s sys 0.000s
I20240502 08:06:16.368657 27659 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.477859 27486 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.479895 27659 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:16.480227 27679 log.cc:924] Time spent T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20240502 08:06:16.480584 27486 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.275805 27534 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
W20240502 08:06:16.482910 27668 log.cc:924] Time spent T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
I20240502 08:06:16.270675 27689 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:16.444290 27663 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:16.386106 27610 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:06:16.388409 26495 rpcz_store.cc:269] 0502 08:06:15.062603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.062821 (+ 218us) service_pool.cc:225] Handling call
0502 08:06:16.348522 (+1285701us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:16.368831 27665 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:16.378463 27369 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:16.356604 27652 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.435395 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.183717 27676 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:16.379360 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 120350us
I20240502 08:06:16.287037 27698 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.450099 27693 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.283735 27671 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.295279 27662 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
W20240502 08:06:16.456815 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 170255us
I20240502 08:06:16.425350 27514 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:16.381227 27697 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.472164 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.483203 27534 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.366876 27691 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:16.484097 26642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 657) took 1115 ms. Trace:
I20240502 08:06:16.489957 26573 raft_consensus.cc:2354] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:16.454250 27576 raft_consensus.cc:686] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:16.477682 27532 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.504666 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:16.368896 27682 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.451191 27611 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.474987 27612 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:16.490986 27610 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:16.460959 27655 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:16.388206 27666 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.483937 27703 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:16.525384 27669 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.523828 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60744 (request call id 390) took 1006 ms. Trace:
I20240502 08:06:16.553735 27691 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:16.463738 27496 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.529366 26571 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:16.503777 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.374272 27687 raft_consensus.cc:504] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.549484 26642 rpcz_store.cc:276] 0502 08:06:15.368471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.798304 (+429833us) service_pool.cc:225] Handling call
0502 08:06:15.798642 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:15.798972 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:15.799001 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:15.856234 (+ 57233us) log.cc:842] Serialized 26 byte log entry
0502 08:06:16.006661 (+150427us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:16.070420 (+ 63759us) log.cc:842] Serialized 14 byte log entry
0502 08:06:16.070624 (+ 204us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:16.070703 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:16.483899 (+413196us) raft_consensus.cc:1671] finished
0502 08:06:16.483916 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:16.484043 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":79047,"raft.queue_time_us":7560,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":27588,"spinlock_wait_cycles":8549504,"thread_start_us":149660,"threads_started":1,"wal-append.queue_time_us":265366}
I20240502 08:06:16.488859 27596 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:16.511438 27527 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.506168 27676 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.451313 27699 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.575938 27612 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:16.526701 27544 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.519749 26616 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:16.574056 27620 consensus_meta.cc:287] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.726s user 0.011s sys 0.000s
I20240502 08:06:16.552843 27600 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.512143 27522 consensus_queue.cc:237] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.594624 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.464823 27667 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:16.599473 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 79843.1us
I20240502 08:06:16.555047 27514 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.607544 26616 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:16.587157 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:16.583664 26501 rpcz_store.cc:276] 0502 08:06:15.517757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.517957 (+ 200us) service_pool.cc:225] Handling call
0502 08:06:15.518350 (+ 393us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:15.527845 (+ 9495us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:15.527876 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:15.552075 (+ 24199us) log.cc:842] Serialized 26 byte log entry
0502 08:06:15.965317 (+413242us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:16.163332 (+198015us) log.cc:842] Serialized 14 byte log entry
0502 08:06:16.238464 (+ 75132us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:16.238579 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:16.523635 (+285056us) raft_consensus.cc:1671] finished
0502 08:06:16.523656 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:16.523775 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13625,"prepare.queue_time_us":194292,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":223,"raft.queue_time_us":170517,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":88375,"thread_start_us":633240,"threads_started":3,"wal-append.queue_time_us":513750}
I20240502 08:06:16.608459 27420 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.616851 27682 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.604068 27620 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.610587 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:16.584981 27655 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.611805 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.506690 27689 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.583281 27369 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.551932 27693 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.938107 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:16.621690 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 596558us
I20240502 08:06:16.604787 27544 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.016027 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.016508 26467 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:16.845301 26574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 777) took 1077 ms. Trace:
I20240502 08:06:17.017239 26574 rpcz_store.cc:276] 0502 08:06:15.768000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.829161 (+ 61161us) service_pool.cc:225] Handling call
0502 08:06:16.845089 (+1015928us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54001ff754. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:16.845225 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245575552}
I20240502 08:06:17.016732 26642 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:17.018772 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.019531 26642 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:16.591956 27669 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.618566 27666 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.772948 27473 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
W20240502 08:06:16.643131 27144 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.577s (SENT) [suppressed 3 similar messages]
I20240502 08:06:16.638990 27592 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.062562 26542 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:16.832465 27684 log.cc:924] Time spent T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20240502 08:06:17.055878 27689 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:17.057304 27669 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.840090 27665 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:17.033082 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.618957 27602 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240502 08:06:17.091303 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 74869.1us
I20240502 08:06:17.092495 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:16.702421 27657 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:17.006130 27533 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.072829 26542 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:16.793202 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:16.618747 27420 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.848865 27652 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.833321 27662 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.997610 27611 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.606595 27696 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:17.033049 27523 raft_consensus.cc:482] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:17.118702 27523 raft_consensus.cc:504] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.119817 26569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 779) took 1349 ms. Trace:
I20240502 08:06:17.120844 27523 leader_election.cc:288] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.700510 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.106691 27565 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
W20240502 08:06:17.088310 26541 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.019s sys 0.033s
I20240502 08:06:16.609349 27600 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:16.624306 27576 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:17.059412 27620 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:16.937239 27637 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:16.893667 27691 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.074440 27653 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:17.089162 27665 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:17.097887 27699 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:16.937666 27655 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.066835 27676 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.084307 27588 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.085517 27496 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.086028 27596 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:17.088469 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:17.087980 27539 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:17.071753 27473 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.091750 27602 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:17.090837 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:17.438592 26650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.076s sys 0.097s
W20240502 08:06:17.168071 26572 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.017s sys 0.021s
W20240502 08:06:17.597786 26650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s user 0.080s sys 0.113s
W20240502 08:06:17.460691 26575 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.000s sys 0.008s
W20240502 08:06:17.600206 26575 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.000s sys 0.010s
W20240502 08:06:17.598459 26572 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.998s user 0.037s sys 0.032s
W20240502 08:06:17.400535 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 789967us
W20240502 08:06:17.596695 26541 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.026s sys 0.065s
I20240502 08:06:17.099601 27666 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:17.120200 26569 rpcz_store.cc:276] 0502 08:06:15.770667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.975897 (+205230us) service_pool.cc:225] Handling call
0502 08:06:15.976223 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:15.990980 (+ 14757us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:15.991012 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:15.991414 (+ 402us) log.cc:842] Serialized 26 byte log entry
0502 08:06:16.177677 (+186263us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:16.197455 (+ 19778us) log.cc:842] Serialized 14 byte log entry
0502 08:06:16.197681 (+ 226us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:16.197779 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:17.119585 (+921806us) raft_consensus.cc:1671] finished
0502 08:06:17.119607 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:17.119739 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14396,"prepare.queue_time_us":19375,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":10917,"raft.queue_time_us":61810,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":10627840,"thread_start_us":156328,"threads_started":1,"wal-append.queue_time_us":362328}
I20240502 08:06:17.612735 26616 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:17.615123 26650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 668) took 1642 ms. Trace:
I20240502 08:06:17.105072 27703 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:17.615656 26650 rpcz_store.cc:276] 0502 08:06:15.963002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.370247 (+407245us) service_pool.cc:225] Handling call
0502 08:06:16.380790 (+ 10543us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:16.381837 (+ 1047us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:16.680241 (+298404us) log.cc:842] Serialized 14 byte log entry
0502 08:06:17.605317 (+925076us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:17.605375 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:17.605415 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:17.605494 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:17.605621 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5462,"prepare.queue_time_us":909948,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":10963,"thread_start_us":1209741,"threads_started":2}
I20240502 08:06:17.406896 27693 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.108271 27544 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.605793 27602 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:17.358741 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.122476 27689 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:17.595665 27707 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:17.596161 27705 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:17.352788 27702 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:17.390573 27490 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:17.512039 27439 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.007s sys 0.013s
W20240502 08:06:17.389147 27610 consensus_meta.cc:287] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 0.809s user 0.006s sys 0.006s
W20240502 08:06:17.419445 26539 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.013s sys 0.046s
I20240502 08:06:17.110430 27696 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:17.107195 27514 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:17.100853 27657 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.216992 27698 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:17.386382 26567 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.017s sys 0.041s
I20240502 08:06:17.641705 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.675616 26650 raft_consensus.cc:2379] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:17.605370 27653 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:17.109189 26649 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:17.646476 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:17.679548 26569 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:17.667312 26539 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.393s user 0.015s sys 0.049s
I20240502 08:06:17.151603 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:17.122498 27565 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:17.611233 27666 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:17.573794 27638 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
W20240502 08:06:17.653690 26467 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.022s sys 0.047s
W20240502 08:06:17.459939 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 816930us
I20240502 08:06:17.672588 27696 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:17.647188 27663 consensus_meta.cc:287] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 1.144s user 0.012s sys 0.000s
I20240502 08:06:17.617112 26499 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:17.611764 27637 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.613481 27539 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.669302 27705 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:17.652391 27689 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:06:17.663969 27439 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s user 0.009s sys 0.016s
I20240502 08:06:17.678436 26650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 436) took 1111 ms. Trace:
I20240502 08:06:17.694355 27638 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.439497 27527 raft_consensus.cc:482] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
W20240502 08:06:17.678247 26567 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.071s user 0.017s sys 0.053s
W20240502 08:06:17.692028 26467 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.022s sys 0.047s
I20240502 08:06:17.636422 27703 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:17.681476 27653 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:17.682502 27514 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:17.666903 27490 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.689064 27666 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:17.670480 27610 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.659073 27707 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.687356 27177 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38552 (request call id 15) took 1247 ms. Trace:
I20240502 08:06:17.687240 27699 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:17.664139 27702 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.690876 26649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 437) took 1113 ms. Trace:
W20240502 08:06:18.102111 27710 log.cc:924] Time spent T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20240502 08:06:17.801169 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:18.103320 26644 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:18.085673 27709 log.cc:924] Time spent T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.390s user 0.000s sys 0.000s
I20240502 08:06:17.721254 27676 raft_consensus.cc:482] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.104607 27676 raft_consensus.cc:504] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.105424 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.105626 26572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 786) took 2265 ms. Trace:
I20240502 08:06:18.066835 27665 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.859709 26569 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
W20240502 08:06:17.801466 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 176270us
W20240502 08:06:18.108125 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 1.01431e+06us
W20240502 08:06:18.014580 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 235715us
I20240502 08:06:18.057253 26574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 788) took 2216 ms. Trace:
W20240502 08:06:18.108829 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 416290us
I20240502 08:06:18.109090 26574 rpcz_store.cc:276] 0502 08:06:15.840983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.063844 (+1222861us) service_pool.cc:225] Handling call
0502 08:06:17.065761 (+ 1917us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:17.069079 (+ 3318us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:17.069118 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:17.077594 (+ 8476us) log.cc:842] Serialized 26 byte log entry
0502 08:06:17.523084 (+445490us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:17.523139 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:17.523234 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.057000 (+533766us) raft_consensus.cc:1671] finished
0502 08:06:18.057028 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.057178 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4012,"raft.queue_time_us":3587,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8140,"thread_start_us":421489,"threads_started":1,"wal-append.queue_time_us":951767}
I20240502 08:06:18.103143 26650 rpcz_store.cc:276] 0502 08:06:16.568548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.637521 (+1068973us) service_pool.cc:225] Handling call
0502 08:06:17.678367 (+ 40846us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:17.706419 27663 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.008103 27727 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:18.106071 26572 rpcz_store.cc:276] 0502 08:06:15.840403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.513652 (+673249us) service_pool.cc:225] Handling call
0502 08:06:16.549636 (+ 35984us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:16.567634 (+ 17998us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:16.567667 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:16.578449 (+ 10782us) log.cc:842] Serialized 26 byte log entry
0502 08:06:17.600825 (+1022376us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:17.639984 (+ 39159us) log.cc:842] Serialized 14 byte log entry
0502 08:06:17.640834 (+ 850us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:17.640924 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.105425 (+464501us) raft_consensus.cc:1671] finished
0502 08:06:18.105444 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.105572 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":466,"prepare.queue_time_us":91654,"prepare.run_cpu_time_us":3266,"prepare.run_wall_time_us":10738,"raft.queue_time_us":35475,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":3144,"thread_start_us":1014279,"threads_started":2,"wal-append.queue_time_us":1076818}
I20240502 08:06:17.705116 27544 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.112772 26616 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.062211 26399 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52756 (request call id 21) took 4056 ms. Trace:
W20240502 08:06:18.084863 26573 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 656) took 2479 ms (client timeout 1886 ms). Trace:
I20240502 08:06:18.116745 26501 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240502 08:06:18.117565 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:18.118217 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
W20240502 08:06:18.121387 26573 rpcz_store.cc:269] 0502 08:06:15.591855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.595475 (+ 3620us) service_pool.cc:225] Handling call
0502 08:06:18.071281 (+2475806us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b540013ecd4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:18.071480 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1685528320}
I20240502 08:06:18.123323 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:18.124352 26467 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:18.185889 26574 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:18.096592 27651 log.cc:924] Time spent T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 2.198s user 0.001s sys 0.010s
W20240502 08:06:18.187191 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 62938.6us
I20240502 08:06:18.122553 26569 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:18.188531 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:18.123885 26395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39272 (request call id 152) took 5272 ms. Trace:
I20240502 08:06:18.191040 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:17.723063 26648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 434) took 1173 ms. Trace:
W20240502 08:06:17.833493 27649 log.cc:924] Time spent T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 2.389s user 0.005s sys 0.000s
W20240502 08:06:17.938817 27612 consensus_meta.cc:287] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 1.329s user 0.012s sys 0.002s
I20240502 08:06:17.707859 27620 consensus_queue.cc:1046] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.416s
I20240502 08:06:18.112084 27689 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:18.103641 26497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 384) took 2629 ms (client timeout 3050 ms). Trace:
I20240502 08:06:17.727690 27539 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.225620 26568 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:17.978791 27600 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.007537 27425 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.106294 26641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 435) took 1557 ms. Trace:
I20240502 08:06:18.186942 27496 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:18.217409 26467 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:18.120054 26399 rpcz_store.cc:276] 0502 08:06:14.006216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:14.008185 (+ 1969us) service_pool.cc:225] Handling call
0502 08:06:18.062086 (+4053901us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:14.609923 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:14.610527 (+ 604us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:14.610556 (+ 29us) write_op.cc:427] Acquired schema lock
0502 08:06:14.610574 (+ 18us) tablet.cc:660] Decoding operations
0502 08:06:14.610895 (+ 321us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:14.610940 (+ 45us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:14.610981 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:06:14.611150 (+ 169us) tablet.cc:699] Row locks acquired
0502 08:06:14.611167 (+ 17us) write_op.cc:260] PREPARE: finished
0502 08:06:14.611293 (+ 126us) write_op.cc:270] Start()
0502 08:06:14.611384 (+ 91us) write_op.cc:276] Timestamp: P: 1714637174611256 usec, L: 0
0502 08:06:14.611404 (+ 20us) op_driver.cc:331] REPLICATION: starting
0502 08:06:14.617882 (+ 6478us) log.cc:842] Serialized 594 byte log entry
0502 08:06:15.618405 (+1000523us) op_driver.cc:444] REPLICATION: finished
0502 08:06:15.886973 (+268568us) write_op.cc:301] APPLY: starting
0502 08:06:15.889202 (+ 2229us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:15.897555 (+ 8353us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:15.918801 (+ 21246us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:15.976408 (+ 57607us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:15.991541 (+ 15133us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:06:16.003534 (+ 11993us) write_op.cc:312] APPLY: finished
0502 08:06:16.026320 (+ 22786us) log.cc:842] Serialized 22 byte log entry
0502 08:06:16.083756 (+ 57436us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:16.089789 (+ 6033us) write_op.cc:446] Released schema lock
0502 08:06:16.093545 (+ 3756us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:16.123015 (+ 29470us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:16.124523 (+ 1508us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":740173,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":242819,"apply.run_cpu_time_us":2931,"apply.run_wall_time_us":275784,"num_ops":1,"prepare.queue_time_us":145612,"prepare.run_cpu_time_us":2319,"prepare.run_wall_time_us":8250,"replication_time_us":990587,"thread_start_us":223334,"threads_started":1}]]}
W20240502 08:06:18.323297 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 106010us
I20240502 08:06:18.005610 27605 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.226424 26572 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:17.985932 27713 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Append to log took a long time: real 0.303s user 0.000s sys 0.000s
W20240502 08:06:17.986666 27715 log.cc:924] Time spent T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.254s user 0.000s sys 0.000s
I20240502 08:06:18.054602 27657 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:17.838698 27691 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.232386 26648 rpcz_store.cc:276] 0502 08:06:16.548111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.151200 (+603089us) service_pool.cc:225] Handling call
0502 08:06:17.721858 (+570658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.302249 26570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51132 (request call id 653) took 2655 ms. Trace:
I20240502 08:06:17.859079 27473 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:17.833565 27565 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:17.994928 27180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:17.818621 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:18.189299 26395 rpcz_store.cc:276] 0502 08:06:12.841230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:12.844103 (+ 2873us) service_pool.cc:225] Handling call
0502 08:06:18.113815 (+5269712us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:13.884037 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:13.985455 (+101418us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:13.985486 (+ 31us) write_op.cc:427] Acquired schema lock
0502 08:06:13.985503 (+ 17us) tablet.cc:660] Decoding operations
0502 08:06:13.985867 (+ 364us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:13.985923 (+ 56us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:13.985956 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:06:13.999211 (+ 13255us) tablet.cc:699] Row locks acquired
0502 08:06:14.002102 (+ 2891us) write_op.cc:260] PREPARE: finished
0502 08:06:14.024743 (+ 22641us) write_op.cc:270] Start()
0502 08:06:14.024870 (+ 127us) write_op.cc:276] Timestamp: P: 1714637174017280 usec, L: 0
0502 08:06:14.024896 (+ 26us) op_driver.cc:331] REPLICATION: starting
0502 08:06:14.025561 (+ 665us) log.cc:842] Serialized 537 byte log entry
0502 08:06:14.656179 (+630618us) op_driver.cc:444] REPLICATION: finished
0502 08:06:15.433612 (+777433us) write_op.cc:301] APPLY: starting
0502 08:06:15.435954 (+ 2342us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:15.456229 (+ 20275us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:15.477219 (+ 20990us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:15.574115 (+ 96896us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:15.593026 (+ 18911us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:06:15.603939 (+ 10913us) write_op.cc:312] APPLY: finished
0502 08:06:15.647618 (+ 43679us) log.cc:842] Serialized 22 byte log entry
0502 08:06:15.727023 (+ 79405us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:15.736122 (+ 9099us) write_op.cc:446] Released schema lock
0502 08:06:15.738059 (+ 1937us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:15.791059 (+ 53000us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:15.804341 (+ 13282us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2867,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":754908,"apply.run_cpu_time_us":4671,"apply.run_wall_time_us":418900,"num_ops":1,"prepare.queue_time_us":438075,"prepare.run_cpu_time_us":121082,"prepare.run_wall_time_us":738389,"raft.queue_time_us":1075611,"raft.run_cpu_time_us":76158,"raft.run_wall_time_us":1238873,"replication_time_us":631188,"thread_start_us":1911982,"threads_started":4,"wal-append.queue_time_us":458128,"wal-append.run_cpu_time_us":19114,"wal-append.run_wall_time_us":2640468}]]}
I20240502 08:06:17.827503 27705 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:18.112536 27703 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:17.936182 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.110308 27514 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:18.086911 26646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 669) took 2123 ms. Trace:
W20240502 08:06:18.405318 27710 log.cc:924] Time spent T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20240502 08:06:18.314105 26614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.000s sys 0.025s
W20240502 08:06:18.420426 26614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.356s user 0.001s sys 0.025s
I20240502 08:06:18.315567 27676 leader_election.cc:288] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.186597 27531 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:18.224576 27487 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:18.233656 27369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.020s sys 0.041s
I20240502 08:06:18.227296 27486 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:18.317823 26574 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:18.123715 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:18.336596 27496 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.234283 27669 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.112541 27527 raft_consensus.cc:504] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.303861 27612 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.116780 27699 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.115810 26649 rpcz_store.cc:276] 0502 08:06:16.573156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.714241 (+141085us) service_pool.cc:225] Handling call
0502 08:06:17.685157 (+970916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.326756 27422 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.299681 27637 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.447839 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60744 (request call id 391) took 2920 ms. Trace:
I20240502 08:06:18.185529 27544 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.224850 27656 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
I20240502 08:06:18.425843 26567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 783) took 2648 ms. Trace:
I20240502 08:06:18.342190 26467 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:06:18.454998 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 113072us
I20240502 08:06:18.448261 26498 rpcz_store.cc:276] 0502 08:06:15.519679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.836653 (+1316974us) service_pool.cc:225] Handling call
0502 08:06:16.837122 (+ 469us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:16.837422 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:16.837458 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:16.837982 (+ 524us) log.cc:842] Serialized 26 byte log entry
0502 08:06:17.443939 (+605957us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:17.685843 (+241904us) log.cc:842] Serialized 14 byte log entry
0502 08:06:17.694853 (+ 9010us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:17.694993 (+ 140us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.439959 (+744966us) raft_consensus.cc:1671] finished
0502 08:06:18.439979 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.440098 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7577,"prepare.queue_time_us":596449,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":1005,"raft.queue_time_us":15977,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":4873,"thread_start_us":613292,"threads_started":2,"wal-append.queue_time_us":736822}
I20240502 08:06:18.371987 27490 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.319028 26641 rpcz_store.cc:276] 0502 08:06:16.548455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.041388 (+1492933us) service_pool.cc:225] Handling call
0502 08:06:18.106236 (+ 64848us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:17.824009 27655 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.318172 27729 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:18.401825 26643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 650) took 3390 ms. Trace:
I20240502 08:06:18.378316 26570 rpcz_store.cc:276] 0502 08:06:15.580117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.580529 (+ 412us) service_pool.cc:225] Handling call
0502 08:06:15.602212 (+ 21683us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:15.602404 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:15.761073 (+158669us) log.cc:842] Serialized 14 byte log entry
0502 08:06:18.013451 (+2252378us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b18002a6e80. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011d3eb5 00007f970a00d939 00007f97011f142b 00007f97011ef82f 00007f97011f1992 00007f9703d9d4eb 00007f9703da2f93 00007f9703da34ab 00007f9704126ada
0502 08:06:18.234011 (+220560us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:18.234083 (+ 72us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.234129 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.234817 (+ 688us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.234958 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35238,"prepare.queue_time_us":151808,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":74564,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7174,"spinlock_wait_cycles":89302912,"thread_start_us":220294,"threads_started":1,"wal-append.queue_time_us":2611835}
I20240502 08:06:18.406379 27530 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:18.407080 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60744 (request call id 386) took 2931 ms. Trace:
I20240502 08:06:18.523228 26496 rpcz_store.cc:276] 0502 08:06:15.475568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.475691 (+ 123us) service_pool.cc:225] Handling call
0502 08:06:15.488806 (+ 13115us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:15.630620 (+141814us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.406815 (+2776195us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b54000fa5d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:18.407001 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216523136}
W20240502 08:06:18.407837 26576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 772) took 2647 ms (client timeout 1904 ms). Trace:
I20240502 08:06:18.334998 27539 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:18.448441 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:18.307085 26497 rpcz_store.cc:269] 0502 08:06:15.474659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.476816 (+ 2157us) service_pool.cc:225] Handling call
0502 08:06:18.103398 (+2626582us) spinlock_profiling.cc:243] Waited 82.7 ms on lock 0x7b54002959d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:18.103561 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":190192000}
I20240502 08:06:18.398779 27735 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.374038 27660 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.224437 27439 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:18.325902 27425 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.405562 27600 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.428552 26569 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:18.428934 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 301879us
W20240502 08:06:18.320263 26500 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.026s sys 0.075s
I20240502 08:06:18.339179 27605 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.407094 27610 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.418818 27514 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:18.430565 27369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.020s sys 0.041s
I20240502 08:06:18.426575 27531 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:18.417475 27178 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
W20240502 08:06:18.574609 26500 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.131s user 0.026s sys 0.077s
W20240502 08:06:18.524310 26576 rpcz_store.cc:269] 0502 08:06:15.760511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.780581 (+ 20070us) service_pool.cc:225] Handling call
0502 08:06:18.407654 (+2627073us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b540023fd54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:18.407764 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202231040}
I20240502 08:06:18.575978 27725 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.576685 27425 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.576522 27725 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:18.576918 26466 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 47401.8us
I20240502 08:06:18.577284 27439 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:18.430521 27676 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.576467 27531 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:18.433436 27733 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:18.578253 27734 log.cc:924] Time spent T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20240502 08:06:18.578331 27733 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
W20240502 08:06:18.579366 27730 log.cc:924] Time spent T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20240502 08:06:18.456092 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.097848 27696 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:18.470713 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 163098us
I20240502 08:06:18.472193 27689 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.187398 27727 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.453598 27637 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.431325 27487 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.440451 27669 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.450652 27422 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:18.593472 27602 consensus_meta.cc:287] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.941s user 0.009s sys 0.003s
I20240502 08:06:18.546023 26574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51122 (request call id 167) took 2463 ms. Trace:
I20240502 08:06:18.522984 26643 rpcz_store.cc:276] 0502 08:06:14.985964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:14.986110 (+ 146us) service_pool.cc:225] Handling call
0502 08:06:15.015805 (+ 29695us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:15.017360 (+ 1555us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:15.017387 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:15.021350 (+ 3963us) log.cc:842] Serialized 26 byte log entry
0502 08:06:15.455430 (+434080us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:15.455485 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:15.455576 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.096056 (+2640480us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b44000232cc. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f9701106685 00007f97011070c2 00007f9703daf441 00007f9703daad4a 00007f9703d9eb1e 00007f9703d9de66 00007f9703d9da88 00007f9703da7481 00007f9703da7439 00007f9703da73c9
0502 08:06:18.338892 (+242836us) raft_consensus.cc:1671] finished
0502 08:06:18.338923 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.375829 (+ 36906us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":666,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2912,"spinlock_wait_cycles":39121280,"thread_start_us":433626,"threads_started":1,"wal-append.queue_time_us":837706}
I20240502 08:06:18.617395 26645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55500 (request call id 640) took 3931 ms. Trace:
I20240502 08:06:18.431835 27486 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:18.625643 27486 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.626062 27620 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.626720 27620 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.454502 26567 rpcz_store.cc:276] 0502 08:06:15.777288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.477232 (+699944us) service_pool.cc:225] Handling call
0502 08:06:16.477573 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:16.496337 (+ 18764us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:16.501445 (+ 5108us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:16.573434 (+ 71989us) log.cc:842] Serialized 26 byte log entry
0502 08:06:18.126291 (+1552857us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.313158 (+186867us) log.cc:842] Serialized 14 byte log entry
0502 08:06:18.403487 (+ 90329us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.408588 (+ 5101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.412808 (+ 4220us) raft_consensus.cc:1671] finished
0502 08:06:18.414553 (+ 1745us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.424028 (+ 9475us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27742,"prepare.queue_time_us":120195,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":101266,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":21053,"thread_start_us":1503292,"threads_started":1,"wal-append.queue_time_us":1211024}
I20240502 08:06:18.628885 27620 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.574589 26616 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.580813 26498 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:18.631129 26498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 845) took 1521 ms (client timeout 1847 ms). Trace:
W20240502 08:06:18.631659 26498 rpcz_store.cc:269] 0502 08:06:17.109792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.455907 (+1346115us) service_pool.cc:225] Handling call
0502 08:06:18.631055 (+175148us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:18.582789 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 109384us
I20240502 08:06:18.522703 27729 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:18.589241 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 832) took 1877 ms. Trace:
I20240502 08:06:18.484505 27665 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.527398 27539 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:06:18.630371 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 55888.3us
I20240502 08:06:18.542157 27740 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.527830 27490 raft_consensus.cc:482] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.418978 26646 rpcz_store.cc:276] 0502 08:06:15.963267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.373055 (+409788us) service_pool.cc:225] Handling call
0502 08:06:16.373419 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:16.621932 (+248513us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:16.623179 (+ 1247us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:16.625881 (+ 2702us) raft_consensus.cc:1631] Marking committed up to 4
0502 08:06:16.633268 (+ 7387us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.086612 (+1453344us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540009bad4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:18.086668 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.086845 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":542189312}
I20240502 08:06:18.605722 26650 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:18.565970 27702 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.115581 27177 rpcz_store.cc:276] 0502 08:06:16.458187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.474936 (+ 16749us) service_pool.cc:225] Handling call
0502 08:06:16.475284 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:16.475490 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 3
0502 08:06:16.885908 (+410418us) log.cc:842] Serialized 14 byte log entry
0502 08:06:17.654572 (+768664us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:17.661941 (+ 7369us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:17.664100 (+ 2159us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:17.670962 (+ 6862us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:17.679152 (+ 8190us) inbound_call.cc:173] Queueing success response
0502 08:06:17.995764 (+316612us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b180026f6e0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011f21b8 00007f97011f5691 00007f97011f5649 00007f97011f55d9 00007f97011f55a1 00007f97011f489d 00007f970a466da4 00007f970a466bd9 00007f97011d4936
Metrics: {"prepare.queue_time_us":877680,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":649375,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":16705,"spinlock_wait_cycles":121070464,"thread_start_us":1057145,"threads_started":3,"wal-append.queue_time_us":769403}
I20240502 08:06:18.609097 27487 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.481257 27698 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:18.545094 27741 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.584403 27439 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:18.588052 26648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 445) took 1552 ms. Trace:
I20240502 08:06:18.619071 26574 rpcz_store.cc:276] 0502 08:06:16.082860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.454076 (+2371216us) service_pool.cc:225] Handling call
0502 08:06:18.545960 (+ 91884us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.626160 27496 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.547613 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.640102 27539 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:18.637580 26500 rpcz_store.cc:276] 0502 08:06:16.711460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.752987 (+ 41527us) service_pool.cc:225] Handling call
0502 08:06:16.753425 (+ 438us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:16.753683 (+ 258us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:17.443476 (+689793us) log.cc:842] Serialized 14 byte log entry
0502 08:06:18.588817 (+1145341us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:18.588884 (+ 67us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.588939 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.589041 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.589173 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35076,"prepare.queue_time_us":1117318,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":21325,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":10845,"thread_start_us":1561776,"threads_started":2,"wal-append.queue_time_us":987542}
I20240502 08:06:18.527693 27530 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.642242 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.641069 27729 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:18.577945 27676 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.609162 27696 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.584183 27600 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.654807 26615 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64; no voters:
I20240502 08:06:18.629552 27620 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.657590 27702 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:18.657544 27741 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.620240 26649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 430) took 2107 ms. Trace:
I20240502 08:06:18.630551 27707 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.618428 27692 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.630132 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.665884 26496 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
W20240502 08:06:18.644058 26568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51132 (request call id 664) took 2644 ms (client timeout 2109 ms). Trace:
I20240502 08:06:18.586714 27660 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.584600 27735 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.651993 27544 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.622170 27602 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.661191 27698 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:18.616062 27637 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:18.676779 27527 leader_election.cc:288] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:18.693176 26496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 843) took 1603 ms (client timeout 2064 ms). Trace:
W20240502 08:06:18.795833 26496 rpcz_store.cc:269] 0502 08:06:17.089430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.524148 (+1434718us) service_pool.cc:225] Handling call
0502 08:06:18.693121 (+168973us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.671111 26641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 433) took 2143 ms. Trace:
I20240502 08:06:18.677726 27733 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:18.739368 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 97799.6us
W20240502 08:06:18.746702 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 57199.3us
I20240502 08:06:18.649642 27490 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.688505 27620 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.692670 27746 raft_consensus.cc:2781] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:06:18.753345 27702 raft_consensus.cc:686] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:18.754271 27605 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.693225 27612 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.755609 27422 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:18.755956 26465 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:58270 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 1 similar messages]
I20240502 08:06:18.617782 26645 rpcz_store.cc:276] 0502 08:06:14.686099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:14.688272 (+ 2173us) service_pool.cc:225] Handling call
0502 08:06:14.706359 (+ 18087us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:14.715955 (+ 9596us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:14.718144 (+ 2189us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:14.785793 (+ 67649us) log.cc:842] Serialized 26 byte log entry
0502 08:06:17.800665 (+3014872us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b44000232cc. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f9701106685 00007f97011070c2 00007f9703daf441 00007f9703daad4a 00007f9703d9eb1e 00007f9703d9de66 00007f9703d9da88 00007f9703da7481 00007f9703da7439 00007f9703da73c9
0502 08:06:18.616884 (+816219us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b180001a020. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f9703dab75c 00007f97011d454c 00007f970a00d939 00007f97011f142b 00007f97011ef82f 00007f97011f1992 00007f9703d9d4eb 00007f9703da2f93 00007f9703da3341
0502 08:06:18.617061 (+ 177us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:18.617097 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.617169 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.617214 (+ 45us) raft_consensus.cc:1671] finished
0502 08:06:18.617228 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.617343 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1909,"raft.queue_time_us":6698,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":13963,"spinlock_wait_cycles":37277952,"thread_start_us":656135,"threads_started":1,"wal-append.queue_time_us":618958}
I20240502 08:06:18.806924 27746 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.814741 27746 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:18.758227 27727 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.685415 27699 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.663183 26648 rpcz_store.cc:276] 0502 08:06:17.033645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.414410 (+1380765us) service_pool.cc:225] Handling call
0502 08:06:18.585967 (+171557us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:18.769887 26568 rpcz_store.cc:269] 0502 08:06:15.999294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.485203 (+485909us) service_pool.cc:225] Handling call
0502 08:06:18.643862 (+2158659us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540023df54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:18.643996 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3806994176}
I20240502 08:06:18.758656 27692 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.801662 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.776022 27707 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.688957 26649 rpcz_store.cc:276] 0502 08:06:16.508309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.455127 (+1946818us) service_pool.cc:225] Handling call
0502 08:06:18.544411 (+ 89284us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:18.568343 (+ 23932us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:18.583420 (+ 15077us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:18.586152 (+ 2732us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.593237 (+ 7085us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.599900 (+ 6663us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.615167 (+ 15267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.646831 27740 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.795132 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 849) took 1111 ms. Trace:
I20240502 08:06:18.818989 26574 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:18.791923 27698 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.815078 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.809975 26646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 441) took 1871 ms. Trace:
I20240502 08:06:18.653328 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60744 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 12)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 855)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 866)
I20240502 08:06:18.829955 26616 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.831049 27727 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:18.828701 26497 rpcz_store.cc:276] 0502 08:06:17.684260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.582510 (+898250us) service_pool.cc:225] Handling call
0502 08:06:18.790841 (+208331us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.850257 27671 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.852993 27729 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.857131 27602 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.859011 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.859968 27702 consensus_queue.cc:237] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.832887 27699 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:18.868396 26616 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.828452 27707 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:18.864005 26644 consensus_meta.cc:287] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.760s user 0.005s sys 0.005s
I20240502 08:06:18.837745 27660 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.840037 27747 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:18.798578 26641 rpcz_store.cc:276] 0502 08:06:16.523784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.541690 (+2017906us) service_pool.cc:225] Handling call
0502 08:06:18.600656 (+ 58966us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:18.631602 (+ 30946us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:18.636890 (+ 5288us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:18.640385 (+ 3495us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.649262 (+ 8877us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.656869 (+ 7607us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.667076 (+ 10207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.837863 26645 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:18.863888 27727 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:18.854691 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.904520 27660 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:18.907480 27741 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.908124 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:18.914757 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:18.870899 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:18.916090 26568 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.878703 27696 raft_consensus.cc:2781] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:18.909686 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60744 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 866)
I20240502 08:06:18.928118 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.930876 27687 leader_election.cc:288] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.931298 27637 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:18.937716 27663 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.935510 27669 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.915062 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.879621 26644 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:18.875430 26497 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:18.905673 27600 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.913930 27727 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.906445 27676 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.941820 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:18.950378 26572 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.942945 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.944343 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.910239 27747 raft_consensus.cc:686] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:18.927708 27660 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.834913 26646 rpcz_store.cc:276] 0502 08:06:16.938232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.666182 (+1727950us) service_pool.cc:225] Handling call
0502 08:06:18.761281 (+ 95099us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:18.790149 (+ 28868us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:18.793437 (+ 3288us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:18.798912 (+ 5475us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.803636 (+ 4724us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.809754 (+ 6118us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.809912 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.946797 26643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 439) took 2339 ms. Trace:
W20240502 08:06:18.938460 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 836) took 1921 ms (client timeout 1820 ms). Trace:
I20240502 08:06:18.941918 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60752 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 864)
I20240502 08:06:18.879247 27699 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.951423 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.939312 26648 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:18.953091 26570 raft_consensus.cc:2379] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:18.962580 26574 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:18.963505 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.963867 27740 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.964336 26570 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:18.962276 27729 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.910853 27486 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.949659 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.966423 26641 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:18.967383 27699 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.967331 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.967866 27486 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.955070 26643 rpcz_store.cc:276] 0502 08:06:16.607062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.617087 (+2010025us) service_pool.cc:225] Handling call
0502 08:06:18.617421 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:18.690774 (+ 73353us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:18.690811 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:18.691324 (+ 513us) log.cc:842] Serialized 26 byte log entry
0502 08:06:18.832386 (+141062us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:18.832415 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.832473 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:18.946578 (+114105us) raft_consensus.cc:1671] finished
0502 08:06:18.946595 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:18.946740 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28174,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10477,"thread_start_us":131601,"threads_started":2,"wal-append.queue_time_us":135043}
I20240502 08:06:18.968192 26641 raft_consensus.cc:2379] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:18.968830 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:18.956280 27743 log.cc:924] Time spent T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20240502 08:06:18.956478 27742 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.093s
I20240502 08:06:18.971709 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.957785 27707 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.965444 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:18.950598 26644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 442) took 2012 ms (client timeout 2517 ms). Trace:
I20240502 08:06:18.972986 26569 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:18.970259 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.973832 26647 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:18.954388 26568 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:18.969300 26572 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:18.957252 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:18.977761 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:18.978405 26572 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:18.955669 27696 raft_consensus.cc:686] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:18.980376 26616 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.956360 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:18.969183 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.954384 27735 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:18.980419 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:18.991415 26647 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55500 (request call id 676) took 2989 ms (client timeout 1523 ms). Trace:
I20240502 08:06:18.968152 27490 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.983908 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:18.991803 26647 rpcz_store.cc:269] 0502 08:06:15.996527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.007112 (+ 10585us) service_pool.cc:225] Handling call
0502 08:06:18.970129 (+2963017us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800204398. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:18.986309 (+ 16180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27610368}
I20240502 08:06:18.993765 26574 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:18.994486 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.959386 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.991948 26616 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.988730 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.997463 26572 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:18.989964 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:18.990175 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:18.998991 26570 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:18.973536 27741 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.980779 27620 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:18.996829 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60752 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 858)
I20240502 08:06:18.997810 26643 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:19.002792 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:18.991842 26568 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:18.995597 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.003793 26647 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:19.004587 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.004776 27490 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:19.005146 26647 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:19.005213 27490 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:18.975347 27477 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:19.006297 27477 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.973238 27637 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:19.000094 26616 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:18.990811 27729 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:18.999192 26569 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:18.991106 26497 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:19.005935 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.007989 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:19.015242 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.015611 26574 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:19.005772 27490 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:19.007668 27741 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:18.998611 27531 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.010089 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:18.959980 26492 rpcz_store.cc:269] 0502 08:06:17.017327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.025708 (+ 8381us) service_pool.cc:225] Handling call
0502 08:06:18.907918 (+1882210us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b5800005198. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:18.938393 (+ 30475us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8006912}
W20240502 08:06:18.976227 26644 rpcz_store.cc:269] 0502 08:06:16.938509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.691098 (+752589us) service_pool.cc:225] Handling call
0502 08:06:18.948290 (+1257192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:19.012300 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:19.016495 26616 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:19.028491 26570 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:19.016964 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:19.014206 26641 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:19.018577 27660 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.012893 27729 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:19.037899 27729 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:19.038658 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:19.039794 27653 consensus_meta.cc:287] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 0.917s user 0.007s sys 0.002s
I20240502 08:06:19.040169 27653 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.012643 27637 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.006348 27746 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:19.043713 27637 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.022609 27602 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:19.022363 27514 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.044495 27473 raft_consensus.cc:3032] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:19.045406 27473 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.045586 27746 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:19.028908 26616 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:19.031407 26573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 789) took 3190 ms. Trace:
I20240502 08:06:19.031612 26568 raft_consensus.cc:2379] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:19.021363 27439 raft_consensus.cc:504] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.049204 26570 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:19.035804 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:19.047566 26573 rpcz_store.cc:276] 0502 08:06:15.841272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.234906 (+2393634us) service_pool.cc:225] Handling call
0502 08:06:18.236434 (+ 1528us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:18.248064 (+ 11630us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:18.248113 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:18.302850 (+ 54737us) log.cc:842] Serialized 26 byte log entry
0502 08:06:18.470225 (+167375us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:18.584676 (+114451us) log.cc:842] Serialized 14 byte log entry
0502 08:06:18.584893 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:18.584979 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:19.031205 (+446226us) raft_consensus.cc:1671] finished
0502 08:06:19.031223 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:19.031343 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38649,"prepare.queue_time_us":113944,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":79935,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":13921,"spinlock_wait_cycles":7617280,"thread_start_us":135633,"threads_started":1,"wal-append.queue_time_us":349983}
I20240502 08:06:19.047920 26616 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:19.036082 26647 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:19.032080 26569 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240502 08:06:19.014379 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60752 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
I20240502 08:06:19.053416 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:19.053575 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.052799 27746 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:19.055271 27746 raft_consensus.cc:686] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.055235 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.056315 26645 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:19.056443 27746 consensus_queue.cc:237] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.057466 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.058984 27477 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.059906 27741 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.061849 27531 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.065070 26616 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:19.011252 27539 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.047628 27620 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:19.072983 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:19.051465 27666 consensus_meta.cc:287] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.925s user 0.008s sys 0.002s
I20240502 08:06:19.044533 27602 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:19.049978 26572 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:19.076965 27620 raft_consensus.cc:686] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.078616 27637 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:19.075143 27473 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.080644 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.072292 27514 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.009075 27747 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.035430 27696 consensus_queue.cc:237] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:19.349089 27607 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.339s user 0.036s sys 0.044s
W20240502 08:06:19.772122 27607 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.762s user 0.039s sys 0.057s
I20240502 08:06:19.227701 27532 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:19.773715 27532 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
W20240502 08:06:19.774816 27490 consensus_meta.cc:287] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.747s user 0.009s sys 0.000s
I20240502 08:06:19.129705 27520 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.775305 27490 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.776609 26643 raft_consensus.cc:2379] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:19.160964 26648 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
W20240502 08:06:19.682103 27727 consensus_meta.cc:287] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.727s user 0.012s sys 0.000s
I20240502 08:06:19.778342 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:19.779750 27473 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:19.272286 26575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 780) took 3499 ms. Trace:
I20240502 08:06:19.187732 27746 raft_consensus.cc:1050] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:19.780449 26575 rpcz_store.cc:276] 0502 08:06:15.773020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.978667 (+205647us) service_pool.cc:225] Handling call
0502 08:06:15.992391 (+ 13724us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:16.589828 (+597437us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:16.589856 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:16.901228 (+311372us) log.cc:842] Serialized 26 byte log entry
0502 08:06:17.600733 (+699505us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:17.600788 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:19.233974 (+1633186us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540013f6d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:19.234072 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:19.234137 (+ 65us) raft_consensus.cc:1671] finished
0502 08:06:19.234153 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:19.272178 (+ 38025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":406712,"raft.queue_time_us":3611,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8518,"spinlock_wait_cycles":1373190016,"thread_start_us":698982,"threads_started":1,"wal-append.queue_time_us":776715,"wal-append.run_cpu_time_us":4766,"wal-append.run_wall_time_us":1064467}
I20240502 08:06:19.111423 27572 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:19.320529 27144 leader_election.cc:334] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.577s (SENT)
I20240502 08:06:19.157145 27620 consensus_queue.cc:237] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.252981 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.787372 26645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 456) took 1012 ms. Trace:
I20240502 08:06:19.787778 26645 rpcz_store.cc:276] 0502 08:06:18.774667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.080356 (+305689us) service_pool.cc:225] Handling call
0502 08:06:19.787311 (+706955us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:19.788115 26642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 425) took 3319 ms (client timeout 2626 ms). Trace:
I20240502 08:06:19.159092 27698 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.788908 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:19.788694 26642 rpcz_store.cc:269] 0502 08:06:16.468971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.030023 (+561052us) service_pool.cc:225] Handling call
0502 08:06:19.252752 (+2222729us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b5800204398. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:19.788027 (+535275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8088320}
I20240502 08:06:19.077986 27741 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:19.082150 27653 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:19.137331 27757 log.cc:924] Time spent T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20240502 08:06:19.175452 27491 raft_consensus.cc:1050] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:19.070601 27750 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:19.835219 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:19.789870 26645 raft_consensus.cc:2396] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 5 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240502 08:06:19.771410 26616 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:19.198252 27530 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.845360 26642 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
W20240502 08:06:19.847769 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 76529.9us
I20240502 08:06:19.084297 27602 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:19.778491 27727 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.853673 26642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 450) took 1170 ms. Trace:
W20240502 08:06:19.854765 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 573832us
W20240502 08:06:19.850261 27758 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.077s user 0.000s sys 0.004s
I20240502 08:06:19.188606 27729 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.850922 27746 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:19.138641 27703 consensus_meta.cc:287] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.000s
W20240502 08:06:19.861418 26571 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.011s sys 0.003s
I20240502 08:06:19.854939 26642 rpcz_store.cc:276] 0502 08:06:18.682861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.847844 (+1164983us) service_pool.cc:225] Handling call
0502 08:06:19.848265 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:19.848483 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:19.848519 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:19.848575 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:19.848617 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:19.848691 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:19.853599 (+ 4908us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:19.855638 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 69949us
W20240502 08:06:19.846355 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 66551.1us
I20240502 08:06:19.086462 27637 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:19.840718 26648 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:06:19.790385 27741 raft_consensus.cc:686] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.863379 26616 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:19.861829 26571 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.524s user 0.012s sys 0.004s
I20240502 08:06:19.084142 27666 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.844926 27750 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:19.863379 26642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 455) took 1163 ms. Trace:
I20240502 08:06:19.856933 27607 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:19.868480 26642 rpcz_store.cc:276] 0502 08:06:18.699541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.862346 (+1162805us) service_pool.cc:225] Handling call
0502 08:06:19.862736 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:19.862972 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:19.863009 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:19.863064 (+ 55us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:19.863111 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:19.863179 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:19.863310 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:19.852466 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 534504us
I20240502 08:06:19.857225 26645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 465) took 1006 ms. Trace:
I20240502 08:06:19.866147 27637 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
W20240502 08:06:19.870095 27762 log.cc:924] Time spent T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.097s user 0.000s sys 0.005s
I20240502 08:06:19.867321 27532 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.867553 27439 leader_election.cc:288] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.869874 27607 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:19.870847 27750 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:19.874941 26649 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:19.865754 27703 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.871701 27725 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.871745 26645 rpcz_store.cc:276] 0502 08:06:18.850528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.788621 (+938093us) service_pool.cc:225] Handling call
0502 08:06:19.855108 (+ 66487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:19.875941 27638 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.882488 27723 raft_consensus.cc:2781] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:19.878638 27607 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:19.893877 26567 raft_consensus.cc:2379] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:19.897433 27723 raft_consensus.cc:686] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.898496 27723 consensus_queue.cc:237] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.899303 27703 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.907622 26571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51146 (request call id 787) took 4067 ms. Trace:
I20240502 08:06:19.910880 27727 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.892519 27741 consensus_queue.cc:237] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.918493 27666 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.920917 26571 rpcz_store.cc:276] 0502 08:06:15.840694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.602281 (+761587us) service_pool.cc:225] Handling call
0502 08:06:16.602621 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:17.031582 (+428961us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:17.031628 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:17.058901 (+ 27273us) log.cc:842] Serialized 26 byte log entry
0502 08:06:19.191435 (+2132534us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b180037b880. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011d3eb5 00007f970a00d939 00007f97011f142b 00007f97011ef82f 00007f97011f1992 00007f9703d9d4eb 00007f9703da2f93 00007f9703da3341 00007f97040f081f
0502 08:06:19.337058 (+145623us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:19.867596 (+530538us) log.cc:842] Serialized 14 byte log entry
0502 08:06:19.872719 (+ 5123us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:19.872783 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:19.907272 (+ 34489us) raft_consensus.cc:1671] finished
0502 08:06:19.907392 (+ 120us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:19.907548 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7531,"prepare.queue_time_us":538702,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":420057,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":11741,"spinlock_wait_cycles":3982592,"thread_start_us":675477,"threads_started":2,"wal-append.queue_time_us":2712745}
I20240502 08:06:19.934796 27729 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.945868 27703 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:19.930833 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 67503.4us
I20240502 08:06:19.965445 27697 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.946605 27703 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:19.966984 26648 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.012280 26648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 467) took 1161 ms. Trace:
I20240502 08:06:20.012617 26648 rpcz_store.cc:276] 0502 08:06:18.850878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.777945 (+927067us) service_pool.cc:225] Handling call
0502 08:06:20.012217 (+234272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:19.981850 27703 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:19.989971 27607 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.016153 27607 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:19.087601 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.018630 27602 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:20.019111 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62425.9us
I20240502 08:06:20.020073 26492 raft_consensus.cc:3037] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.024333 26643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 470) took 1172 ms. Trace:
I20240502 08:06:20.024818 26643 rpcz_store.cc:276] 0502 08:06:18.851394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.002575 (+151181us) service_pool.cc:225] Handling call
0502 08:06:20.024078 (+1021503us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54001cbbd4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.024261 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1777850240}
I20240502 08:06:20.032860 26616 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.037256 27746 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:20.037686 27746 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:20.038141 27746 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:20.042196 26492 raft_consensus.cc:2454] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:20.032107 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 170094us
I20240502 08:06:20.042857 27723 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:20.047896 26615 leader_election.cc:334] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.595s (SENT)
I20240502 08:06:20.048496 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:20.049825 27473 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:20.050333 27177 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:20.042694 27602 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:20.052878 27770 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:06:20.047358 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859) took 1947 ms (client timeout 1970 ms). Trace:
I20240502 08:06:20.053951 27746 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.053508 27750 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.047433 27620 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:20.056694 27637 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.067427 27723 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.072353 27723 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:20.069301 27620 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.073942 27746 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:20.067219 26492 rpcz_store.cc:269] 0502 08:06:18.100156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.050811 (+950655us) service_pool.cc:225] Handling call
0502 08:06:20.046707 (+995896us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.083022 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.085909 26492 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:20.087478 27637 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:20.088227 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865) took 1927 ms (client timeout 1927 ms). Trace:
I20240502 08:06:20.087656 27620 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:20.088718 26492 rpcz_store.cc:269] 0502 08:06:18.160595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.082661 (+1922066us) service_pool.cc:225] Handling call
0502 08:06:20.088159 (+ 5498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.090358 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.091663 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863) took 1990 ms (client timeout 2001 ms). Trace:
W20240502 08:06:20.092094 26492 rpcz_store.cc:269] 0502 08:06:18.101450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.090003 (+1988553us) service_pool.cc:225] Handling call
0502 08:06:20.091607 (+ 1604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.093187 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.094090 26492 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
W20240502 08:06:20.094599 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839) took 3027 ms (client timeout 3063 ms). Trace:
W20240502 08:06:20.095743 26492 rpcz_store.cc:269] 0502 08:06:17.067760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.092865 (+3025105us) service_pool.cc:225] Handling call
0502 08:06:20.094542 (+ 1677us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.096868 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.097571 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860) took 1997 ms (client timeout 2037 ms). Trace:
W20240502 08:06:20.097986 26492 rpcz_store.cc:269] 0502 08:06:18.100497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.096601 (+1996104us) service_pool.cc:225] Handling call
0502 08:06:20.097526 (+ 925us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.098776 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.099350 26492 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.105585 26492 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.071817 27750 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:20.106689 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854) took 2064 ms (client timeout 2108 ms). Trace:
I20240502 08:06:20.137647 26567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51146 (request call id 806) took 1263 ms. Trace:
I20240502 08:06:20.244488 26567 rpcz_store.cc:276] 0502 08:06:18.874457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.878449 (+ 3992us) service_pool.cc:225] Handling call
0502 08:06:20.137412 (+1258963us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54002a0654. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.137570 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2173372672}
W20240502 08:06:20.212342 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 837) took 3182 ms (client timeout 1595 ms). Trace:
W20240502 08:06:20.245895 26493 rpcz_store.cc:269] 0502 08:06:17.030271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.033395 (+ 3124us) service_pool.cc:225] Handling call
0502 08:06:20.212066 (+3178671us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001180d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.212254 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2823980416}
I20240502 08:06:19.231127 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.247131 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419) took 1623 ms (client timeout 1544 ms). Trace:
I20240502 08:06:20.247292 26500 raft_consensus.cc:2362] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4. Re-sending same reply.
W20240502 08:06:20.247565 26493 rpcz_store.cc:269] 0502 08:06:18.624125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.246897 (+1622772us) service_pool.cc:212] Skipping call since client already timed out
0502 08:06:20.247041 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240502 08:06:20.247943 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 12) took 2128 ms (client timeout 1577 ms). Trace:
W20240502 08:06:20.239665 26495 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.129s user 0.019s sys 0.014s
W20240502 08:06:20.248725 26495 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.138s user 0.019s sys 0.014s
W20240502 08:06:20.248551 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842) took 3161 ms (client timeout 3124 ms). Trace:
W20240502 08:06:20.249820 26493 rpcz_store.cc:269] 0502 08:06:17.087274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.248355 (+3161081us) service_pool.cc:212] Skipping call since client already timed out
0502 08:06:20.248477 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240502 08:06:20.252281 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.248376 26500 rpcz_store.cc:269] 0502 08:06:18.119709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.230836 (+1111127us) service_pool.cc:225] Handling call
0502 08:06:20.247878 (+1017042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.255782 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.256423 26500 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:20.257277 26498 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:20.262527 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402) took 1661 ms (client timeout 1694 ms). Trace:
W20240502 08:06:20.263111 26500 rpcz_store.cc:269] 0502 08:06:18.601540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.255514 (+1653974us) service_pool.cc:225] Handling call
0502 08:06:20.262466 (+ 6952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.263937 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.264689 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867) took 2066 ms (client timeout 2110 ms). Trace:
W20240502 08:06:20.265009 26500 rpcz_store.cc:269] 0502 08:06:18.198417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.263709 (+2065292us) service_pool.cc:225] Handling call
0502 08:06:20.264638 (+ 929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.265830 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.266433 26500 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
W20240502 08:06:20.266875 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403) took 1665 ms (client timeout 1749 ms). Trace:
I20240502 08:06:20.258072 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.258509 26644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 466) took 1407 ms. Trace:
W20240502 08:06:20.267189 26500 rpcz_store.cc:269] 0502 08:06:18.601859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.265575 (+1663716us) service_pool.cc:225] Handling call
0502 08:06:20.266828 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.267421 26644 rpcz_store.cc:276] 0502 08:06:18.850726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.038408 (+187682us) service_pool.cc:225] Handling call
0502 08:06:20.258298 (+1219890us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001cc5d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.258450 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2308765440}
W20240502 08:06:20.267598 26649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 447) took 1585 ms (client timeout 1736 ms). Trace:
I20240502 08:06:20.268481 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.269205 26649 rpcz_store.cc:269] 0502 08:06:18.681834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.832951 (+151117us) service_pool.cc:225] Handling call
0502 08:06:20.267420 (+1434469us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54000bcf54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.267544 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2181365632}
W20240502 08:06:20.267838 26492 rpcz_store.cc:269] 0502 08:06:18.042475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.098523 (+2056048us) service_pool.cc:225] Handling call
0502 08:06:20.106629 (+ 8106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:20.270453 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401) took 1669 ms (client timeout 1706 ms). Trace:
W20240502 08:06:20.270893 26493 rpcz_store.cc:269] 0502 08:06:18.601171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.257756 (+1656585us) service_pool.cc:225] Handling call
0502 08:06:20.270393 (+ 12637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.271817 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.272456 26493 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
W20240502 08:06:20.272943 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418) took 1649 ms (client timeout 1736 ms). Trace:
W20240502 08:06:20.273315 26493 rpcz_store.cc:269] 0502 08:06:18.623835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.271550 (+1647715us) service_pool.cc:225] Handling call
0502 08:06:20.272894 (+ 1344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.277534 26498 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
W20240502 08:06:20.278501 26498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 855) took 2179 ms (client timeout 1704 ms). Trace:
W20240502 08:06:20.278888 26498 rpcz_store.cc:269] 0502 08:06:18.098731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.229359 (+1130628us) service_pool.cc:225] Handling call
0502 08:06:20.278430 (+1049071us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.279799 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.280438 26498 raft_consensus.cc:3037] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.283510 27490 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:20.285960 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.287099 26493 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
W20240502 08:06:20.287624 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409) took 1683 ms (client timeout 1765 ms). Trace:
W20240502 08:06:20.288017 26493 rpcz_store.cc:269] 0502 08:06:18.603733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.273926 (+1670193us) service_pool.cc:225] Handling call
0502 08:06:20.287565 (+ 13639us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.289005 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.289824 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424) took 1656 ms (client timeout 1928 ms). Trace:
W20240502 08:06:20.290233 26493 rpcz_store.cc:269] 0502 08:06:18.632903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.288703 (+1655800us) service_pool.cc:225] Handling call
0502 08:06:20.289772 (+ 1069us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.294894 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.269655 26500 raft_consensus.cc:2396] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240502 08:06:20.305886 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 835) took 3366 ms. Trace:
W20240502 08:06:20.314666 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421) took 1682 ms (client timeout 1929 ms). Trace:
W20240502 08:06:20.321287 26492 rpcz_store.cc:269] 0502 08:06:18.632320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.294579 (+1662259us) service_pool.cc:225] Handling call
0502 08:06:20.314593 (+ 20014us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.322433 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.323321 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 429) took 1447 ms (client timeout 1706 ms). Trace:
W20240502 08:06:20.323719 26492 rpcz_store.cc:269] 0502 08:06:18.876107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.322116 (+1446009us) service_pool.cc:225] Handling call
0502 08:06:20.323262 (+ 1146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.324563 26501 rpcz_store.cc:276] 0502 08:06:16.939337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.939549 (+ 212us) service_pool.cc:225] Handling call
0502 08:06:17.020736 (+ 81187us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:18.121078 (+1100342us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:20.305565 (+2184487us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000b1b54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:20.305817 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2519980928}
I20240502 08:06:20.325910 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.314622 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11) took 2195 ms (client timeout 2237 ms). Trace:
W20240502 08:06:20.327091 26500 rpcz_store.cc:269] 0502 08:06:18.119392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.268258 (+2148866us) service_pool.cc:225] Handling call
0502 08:06:20.314564 (+ 46306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.328011 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.328819 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415) took 1706 ms (client timeout 2021 ms). Trace:
W20240502 08:06:20.329192 26500 rpcz_store.cc:269] 0502 08:06:18.622744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.327769 (+1705025us) service_pool.cc:225] Handling call
0502 08:06:20.328769 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.330063 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.330706 26498 raft_consensus.cc:2454] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
W20240502 08:06:20.359925 26498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 430) took 1420 ms (client timeout 1535 ms). Trace:
W20240502 08:06:20.360357 26498 rpcz_store.cc:269] 0502 08:06:18.941337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.279545 (+1338208us) service_pool.cc:225] Handling call
0502 08:06:20.359861 (+ 80316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.360689 27689 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:20.330847 26501 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
W20240502 08:06:20.331179 26647 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 449) took 1648 ms (client timeout 2170 ms). Trace:
W20240502 08:06:20.361896 26501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406) took 1759 ms (client timeout 2003 ms). Trace:
W20240502 08:06:20.362620 26501 rpcz_store.cc:269] 0502 08:06:18.602806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.325653 (+1722847us) service_pool.cc:225] Handling call
0502 08:06:20.361841 (+ 36188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.363441 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.364177 26501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420) took 1739 ms (client timeout 2082 ms). Trace:
W20240502 08:06:20.364466 26501 rpcz_store.cc:269] 0502 08:06:18.624406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.363214 (+1738808us) service_pool.cc:225] Handling call
0502 08:06:20.364131 (+ 917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.365192 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.365700 26501 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:20.331441 26500 raft_consensus.cc:2454] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
W20240502 08:06:20.366268 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411) took 1754 ms (client timeout 2070 ms). Trace:
W20240502 08:06:20.366577 26500 rpcz_store.cc:269] 0502 08:06:18.611570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.329816 (+1718246us) service_pool.cc:225] Handling call
0502 08:06:20.366219 (+ 36403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.367355 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.368124 26500 raft_consensus.cc:2396] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20240502 08:06:20.368526 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416) took 1745 ms (client timeout 2151 ms). Trace:
W20240502 08:06:20.368834 26500 rpcz_store.cc:269] 0502 08:06:18.623215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.367141 (+1743926us) service_pool.cc:225] Handling call
0502 08:06:20.368476 (+ 1335us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:20.369603 27491 raft_consensus.cc:1054] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Unable to promote non-voter 5216c4a3a33f47ac88fa35704487f77e: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:20.369649 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.370915 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422) took 1738 ms (client timeout 2170 ms). Trace:
W20240502 08:06:20.371275 26500 rpcz_store.cc:269] 0502 08:06:18.632505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.369423 (+1736918us) service_pool.cc:225] Handling call
0502 08:06:20.370863 (+ 1440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.372116 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.372694 26500 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:20.373148 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856) took 2274 ms (client timeout 2752 ms). Trace:
W20240502 08:06:20.373462 26500 rpcz_store.cc:269] 0502 08:06:18.099208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.371871 (+2272663us) service_pool.cc:225] Handling call
0502 08:06:20.373103 (+ 1232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.374307 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.375171 26500 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
W20240502 08:06:20.375545 26500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404) took 1773 ms (client timeout 2251 ms). Trace:
W20240502 08:06:20.375846 26500 rpcz_store.cc:269] 0502 08:06:18.602221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.374069 (+1771848us) service_pool.cc:225] Handling call
0502 08:06:20.375500 (+ 1431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.361297 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.382464 26498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 425) took 1513 ms (client timeout 1828 ms). Trace:
I20240502 08:06:20.376631 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.384652 26500 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:20.385130 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 433) took 1371 ms. Trace:
I20240502 08:06:20.385423 26500 rpcz_store.cc:276] 0502 08:06:19.013921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.376417 (+1362496us) service_pool.cc:225] Handling call
0502 08:06:20.385079 (+ 8662us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.386391 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.386955 26500 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:20.388412 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 435) took 1293 ms. Trace:
I20240502 08:06:20.388695 26500 rpcz_store.cc:276] 0502 08:06:19.095142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.386127 (+1290985us) service_pool.cc:225] Handling call
0502 08:06:20.388361 (+ 2234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.389508 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.382853 26498 rpcz_store.cc:269] 0502 08:06:18.869493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.361021 (+1491528us) service_pool.cc:225] Handling call
0502 08:06:20.382405 (+ 21384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.390985 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.391880 26498 raft_consensus.cc:2396] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20240502 08:06:20.392261 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407) took 1789 ms. Trace:
I20240502 08:06:20.392550 26498 rpcz_store.cc:276] 0502 08:06:18.603140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.390743 (+1787603us) service_pool.cc:225] Handling call
0502 08:06:20.392213 (+ 1470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.393365 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.334245 26497 consensus_meta.cc:287] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 1.317s user 0.015s sys 0.000s
I20240502 08:06:20.390015 26500 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.394376 26497 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
W20240502 08:06:20.394915 26497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 862) took 2293 ms (client timeout 1818 ms). Trace:
I20240502 08:06:20.343255 26499 raft_consensus.cc:2354] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:20.356740 26398 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.395531 26642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55494 (request call id 106) took 1543 ms. Trace:
I20240502 08:06:20.395942 26642 rpcz_store.cc:276] 0502 08:06:18.852177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.871660 (+1019483us) service_pool.cc:225] Handling call
0502 08:06:20.394840 (+523180us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b540009d3d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a1d30 00007f970412517a 00007f9709062a1e 00007f9702755817 00007f97027557a2 00007f97027556d1 00007f970275564c 00007f9702754902 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.395459 (+ 619us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85924096}
I20240502 08:06:20.396551 26398 catalog_manager.cc:5484] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
W20240502 08:06:20.397799 26499 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 844) took 3306 ms (client timeout 1865 ms). Trace:
I20240502 08:06:20.398175 26398 catalog_manager.cc:5484] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
W20240502 08:06:20.399150 26499 rpcz_store.cc:269] 0502 08:06:17.091457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:17.825008 (+733551us) service_pool.cc:225] Handling call
0502 08:06:20.397580 (+2572572us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400117bd4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.397727 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4428248832}
I20240502 08:06:20.400353 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.399819 26398 catalog_manager.cc:5484] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.401229 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 432) took 1404 ms. Trace:
I20240502 08:06:20.331229 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.402369 26492 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:20.402544 27692 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:20.402926 26492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405) took 1800 ms (client timeout 1984 ms). Trace:
W20240502 08:06:20.395277 26497 rpcz_store.cc:269] 0502 08:06:18.101123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.956118 (+854995us) service_pool.cc:225] Handling call
0502 08:06:20.394856 (+1438738us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:20.403370 26492 rpcz_store.cc:269] 0502 08:06:18.602518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.324402 (+1721884us) service_pool.cc:225] Handling call
0502 08:06:20.402869 (+ 78467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.404007 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.404402 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.405007 26501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 431) took 1443 ms (client timeout 1749 ms). Trace:
I20240502 08:06:20.326309 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.361840 26647 rpcz_store.cc:269] 0502 08:06:18.682548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.015046 (+332498us) service_pool.cc:225] Handling call
0502 08:06:20.331026 (+1315980us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b54003903d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.331129 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46016384}
I20240502 08:06:20.401604 26499 rpcz_store.cc:276] 0502 08:06:18.996490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.400073 (+1403583us) service_pool.cc:225] Handling call
0502 08:06:20.401171 (+ 1098us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.406251 26493 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
W20240502 08:06:20.406873 26493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 428) took 1535 ms (client timeout 1694 ms). Trace:
I20240502 08:06:20.407086 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.407271 26493 rpcz_store.cc:269] 0502 08:06:18.871195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.326002 (+1454807us) service_pool.cc:225] Handling call
0502 08:06:20.406821 (+ 80819us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.407922 26499 raft_consensus.cc:2379] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.405090 26492 raft_consensus.cc:3037] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.408360 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 427) took 1537 ms. Trace:
I20240502 08:06:20.408241 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:19.230234 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.409317 26493 raft_consensus.cc:2396] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:20.409801 26493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410) took 1805 ms. Trace:
I20240502 08:06:20.410248 26493 rpcz_store.cc:276] 0502 08:06:18.607553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.407979 (+1800426us) service_pool.cc:225] Handling call
0502 08:06:20.409746 (+ 1767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:18.818605 26465 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:58270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 402)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58270 (request call id 11)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 411)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 404)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 866)
W20240502 08:06:20.409339 26496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 868) took 2151 ms (client timeout 1595 ms). Trace:
W20240502 08:06:20.411881 26496 rpcz_store.cc:269] 0502 08:06:18.258473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:19.229841 (+971368us) service_pool.cc:225] Handling call
0502 08:06:20.409276 (+1179435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.412683 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.413473 26496 raft_consensus.cc:2396] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:20.417557 27539 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:20.413851 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 434) took 1337 ms. Trace:
W20240502 08:06:20.406090 26501 rpcz_store.cc:269] 0502 08:06:18.961213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.364999 (+1403786us) service_pool.cc:225] Handling call
0502 08:06:20.404941 (+ 39942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.418234 26496 rpcz_store.cc:276] 0502 08:06:19.076565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.412473 (+1335908us) service_pool.cc:225] Handling call
0502 08:06:20.413801 (+ 1328us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.418967 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.393929 26498 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:20.405687 26497 raft_consensus.cc:2379] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.420612 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 423) took 1787 ms. Trace:
I20240502 08:06:20.420785 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 408) took 1817 ms. Trace:
I20240502 08:06:20.421213 26497 rpcz_store.cc:276] 0502 08:06:18.603444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.403732 (+1800288us) service_pool.cc:225] Handling call
0502 08:06:20.420732 (+ 17000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.401527 26398 catalog_manager.cc:5484] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.423568 26398 catalog_manager.cc:5484] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.411154 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.429298 26495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 833) took 3651 ms. Trace:
I20240502 08:06:20.420979 26498 rpcz_store.cc:276] 0502 08:06:18.632679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.393143 (+1760464us) service_pool.cc:225] Handling call
0502 08:06:20.420561 (+ 27418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.434955 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.434409 26495 rpcz_store.cc:276] 0502 08:06:16.778340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:16.993294 (+214954us) service_pool.cc:225] Handling call
0502 08:06:19.075323 (+2082029us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:19.084805 (+ 9482us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:20.249239 (+1164434us) log.cc:842] Serialized 14 byte log entry
0502 08:06:20.261008 (+ 11769us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:20.261068 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:20.261132 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:20.261240 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:20.429080 (+167840us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54000fbed0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411bc59 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.429232 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":690426,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":94,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":4615135488,"thread_start_us":1149196,"threads_started":2,"wal-append.queue_time_us":11450}
I20240502 08:06:20.436298 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.437040 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.437496 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.439131 26493 raft_consensus.cc:2454] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 1.
I20240502 08:06:20.438690 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.440541 26541 leader_election.cc:334] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.694s (SENT)
I20240502 08:06:20.440088 26493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 417) took 1816 ms. Trace:
I20240502 08:06:20.440991 26541 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:20.441685 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.442909 27653 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:20.443645 26541 leader_election.cc:334] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.706s (SENT)
I20240502 08:06:20.447657 26541 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:20.449746 26541 leader_election.cc:334] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.749s (SENT)
I20240502 08:06:20.450174 26541 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:20.450932 27739 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:20.451542 27739 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:20.453554 27702 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
W20240502 08:06:20.453755 26541 leader_election.cc:334] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.765s (SENT)
I20240502 08:06:20.454226 27702 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:20.441273 26493 rpcz_store.cc:276] 0502 08:06:18.623545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.410872 (+1787327us) service_pool.cc:225] Handling call
0502 08:06:20.440028 (+ 29156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.457168 27702 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:20.457406 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.459125 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.460554 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.461745 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.463333 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:20.463917 26645 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:20.464548 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.465330 26649 raft_consensus.cc:2379] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.466077 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.466795 26649 raft_consensus.cc:2379] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.467536 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:20.468047 26649 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.469893 27750 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:20.470554 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:20.470417 27750 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.470914 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:20.471343 26567 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.471609 26643 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.472481 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.473227 26643 raft_consensus.cc:2379] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.474359 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.474987 26648 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:20.475396 26495 raft_consensus.cc:2362] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4. Re-sending same reply.
I20240502 08:06:20.475821 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.476986 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.478564 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:20.478825 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.479137 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.479243 26576 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.464126 26642 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:20.480088 26647 raft_consensus.cc:2379] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 2: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.479868 26573 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:20.473882 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.481477 26643 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.470363 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.482952 26644 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.483704 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.484354 26642 raft_consensus.cc:2362] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:20.485106 26498 raft_consensus.cc:2454] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.487637 26500 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.488025 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 413) took 1876 ms. Trace:
I20240502 08:06:20.488229 26500 rpcz_store.cc:276] 0502 08:06:18.611914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.389272 (+1777358us) service_pool.cc:225] Handling call
0502 08:06:20.487971 (+ 98699us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.489053 26398 catalog_manager.cc:5484] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.490674 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.490839 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.491469 26573 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.491654 26643 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.433502 26492 raft_consensus.cc:2454] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.493814 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.496366 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.496940 26572 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.494668 26647 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:20.498320 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:20.472497 27750 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:20.498776 26642 raft_consensus.cc:2362] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:20.510145 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.629817 26616 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.533483 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.569190 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.531216 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.644764 26498 raft_consensus.cc:2354] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3.
I20240502 08:06:20.643317 26576 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.646832 26647 raft_consensus.cc:2379] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:20.645509 26500 raft_consensus.cc:2362] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3. Re-sending same reply.
I20240502 08:06:20.622906 26573 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:20.666586 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:20.675038 26398 catalog_manager.cc:5484] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.637383 26572 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:20.698796 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:20.667420 26573 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.723562 26398 catalog_manager.cc:5484] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.696265 26616 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.723626 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.728447 26573 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:20.732782 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.733783 26500 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:20.739157 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.739974 26498 raft_consensus.cc:2454] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 1.
I20240502 08:06:20.741048 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.742376 27680 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:20.742915 27680 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:20.743374 27680 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.541085 26492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 414) took 1918 ms. Trace:
I20240502 08:06:20.746313 26492 rpcz_store.cc:276] 0502 08:06:18.622441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.404145 (+1781704us) service_pool.cc:225] Handling call
0502 08:06:20.509408 (+105263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.747819 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.741737 26498 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:20.510210 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:20.748579 26500 raft_consensus.cc:2362] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4. Re-sending same reply.
I20240502 08:06:20.674777 27751 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:20.760367 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.762745 26648 raft_consensus.cc:2362] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:20.763134 27751 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:20.765225 27751 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.770586 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.763727 26649 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:20.755597 26398 catalog_manager.cc:5484] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:20.786875 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.770610 26500 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:20.811309 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.826547 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 129219us
I20240502 08:06:20.832866 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:20.834540 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
W20240502 08:06:20.837930 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 423614us
I20240502 08:06:20.902189 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:20.903080 26645 raft_consensus.cc:2454] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:20.545950 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 840) took 3469 ms. Trace:
I20240502 08:06:20.490748 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.904912 26498 raft_consensus.cc:2362] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3. Re-sending same reply.
I20240502 08:06:20.905643 26500 raft_consensus.cc:2379] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:20.493683 26499 rpcz_store.cc:276] 0502 08:06:18.870854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.406815 (+1535961us) service_pool.cc:225] Handling call
0502 08:06:20.408304 (+ 1489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:20.906200 26572 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
W20240502 08:06:20.908113 26541 leader_election.cc:334] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.984s (SENT)
W20240502 08:06:20.914680 26541 leader_election.cc:334] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.003s (SENT)
I20240502 08:06:20.914837 26573 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
W20240502 08:06:20.915261 26541 leader_election.cc:334] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.251s (SENT)
I20240502 08:06:20.915710 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.908437 26616 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:20.545279 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 91803.7us
I20240502 08:06:20.916303 26573 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:20.917327 26615 leader_election.cc:334] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.606s (SENT)
I20240502 08:06:20.918694 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:20.919299 26572 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.909487 26496 rpcz_store.cc:276] 0502 08:06:17.076593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.418817 (+3342224us) service_pool.cc:225] Handling call
0502 08:06:20.419053 (+ 236us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:20.419226 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:20.446244 (+ 27018us) log.cc:842] Serialized 14 byte log entry
0502 08:06:20.545609 (+ 99365us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:20.545658 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:20.545701 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:20.545766 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:20.545888 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15366,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":24056,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7407,"thread_start_us":124795,"threads_started":2}
I20240502 08:06:20.921432 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:20.922489 26615 leader_election.cc:334] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.643s (SENT)
I20240502 08:06:20.922552 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.923002 26573 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:20.904493 26576 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:20.923000 26615 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:20.918268 26641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 446) took 2230 ms (client timeout 1929 ms). Trace:
I20240502 08:06:20.927366 26616 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.927237 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.937896 26500 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:20.938997 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.938939 26615 leader_election.cc:334] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.704s (SENT)
I20240502 08:06:20.941282 26616 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.942462 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:20.936995 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 50417.6us
W20240502 08:06:20.934917 26641 rpcz_store.cc:269] 0502 08:06:18.681565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.968913 (+287348us) service_pool.cc:225] Handling call
0502 08:06:20.837524 (+1868611us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b540038fed4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:20.910803 (+ 73279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55732992}
W20240502 08:06:19.090220 26494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60744 (request call id 387) took 3614 ms (client timeout 2292 ms). Trace:
I20240502 08:06:20.949195 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:20.925393 27653 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:20.925348 27422 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:20.927650 26493 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:20.947398 26494 rpcz_store.cc:269] 0502 08:06:15.489160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:15.491193 (+ 2033us) service_pool.cc:225] Handling call
0502 08:06:19.068158 (+3576965us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b54000fbed4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:19.086579 (+ 18421us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1020623104}
I20240502 08:06:20.938889 27750 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:20.950452 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.941967 27702 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:20.971943 26496 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:20.991019 27787 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:20.993141 27750 raft_consensus.cc:686] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:20.958912 27692 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:20.993840 27692 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:20.994287 27692 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:20.994172 27750 consensus_queue.cc:237] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.959408 27620 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:20.998159 27620 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:20.934583 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:20.955081 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.121665 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 841) took 4037 ms. Trace:
W20240502 08:06:21.119669 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 159343us
I20240502 08:06:21.130189 26572 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.068486 26576 raft_consensus.cc:1261] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:21.101038 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:21.131829 26575 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:21.102456 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:21.113744 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.134248 26568 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.130538 26501 rpcz_store.cc:276] 0502 08:06:17.083942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.419955 (+3336013us) service_pool.cc:225] Handling call
0502 08:06:20.421295 (+ 1340us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:20.421456 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:20.421481 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:20.421519 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:20.421548 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:20.421598 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.121428 (+699830us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5400386150. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411bc59 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:21.121592 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2300672}
W20240502 08:06:21.137385 26615 leader_election.cc:334] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.820s (SENT)
I20240502 08:06:21.133342 26571 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:21.003295 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.144311 26574 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:21.154583 26571 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:21.157668 26615 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:21.102686 27786 log.cc:924] Time spent T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
W20240502 08:06:21.099277 26522 debug-util.cc:398] Leaking SignalData structure 0x7b08001dcd40 after lost signal to thread 25355
W20240502 08:06:21.095012 26542 connection.cc:232] RPC call timeout handler was delayed by -1.0258190119998289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20240502 08:06:21.114692 27785 log.cc:924] Time spent T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20240502 08:06:20.998276 27702 raft_consensus.cc:686] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.169479 26575 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:21.177137 26576 raft_consensus.cc:1261] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:21.177383 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.186092 26570 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:21.178674 26572 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:21.117939 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:21.115829 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.099102 27667 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:21.104219 27680 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.001873 27698 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:21.189832 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.189925 26616 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:20.961575 27653 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.160251 26492 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.192400 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:21.196062 26541 leader_election.cc:334] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.070s (SENT)
I20240502 08:06:21.202579 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.202896 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.170707 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 846) took 3556 ms. Trace:
I20240502 08:06:21.206017 26498 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.207131 26616 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:21.204460 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 67112.4us
I20240502 08:06:21.209484 26570 raft_consensus.cc:2454] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
W20240502 08:06:21.213438 26615 leader_election.cc:334] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.847s (SENT)
I20240502 08:06:21.211477 26616 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:21.216712 26650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 440) took 4278 ms (client timeout 2964 ms). Trace:
W20240502 08:06:21.223214 26650 rpcz_store.cc:269] 0502 08:06:16.937873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.117337 (+1179464us) service_pool.cc:225] Handling call
0502 08:06:21.216472 (+3099135us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540009b0d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97041204fe 00007f970905e9d7 00007f9702742c14 00007f9702742ba2 00007f9702742ad1 00007f9702742a4c 00007f9702741d02 00007f9702543bcc 00007f9702543006 00007f97025429af
0502 08:06:21.216633 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402740096}
I20240502 08:06:21.192008 27596 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:21.192358 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.169862 27751 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.219334 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.227587 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:21.210711 27692 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.220270 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:21.215901 27607 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:21.220350 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.192648 26646 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:21.205734 27752 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.228353 26569 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:21.228170 26498 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:21.228797 26568 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:21.230557 26615 leader_election.cc:334] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.865s (SENT)
I20240502 08:06:21.247463 26615 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:21.218266 26574 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.218058 27746 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
W20240502 08:06:21.211026 27144 leader_election.cc:334] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.237s (SENT)
I20240502 08:06:21.247493 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:21.266697 26522 debug-util.cc:398] Leaking SignalData structure 0x7b080019c480 after lost signal to thread 26433
I20240502 08:06:21.267185 26498 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.266881 27144 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:21.268002 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 62519.7us
I20240502 08:06:21.268524 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 848) took 3603 ms. Trace:
I20240502 08:06:21.268877 26498 rpcz_store.cc:276] 0502 08:06:17.665319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.267753 (+3602434us) service_pool.cc:225] Handling call
0502 08:06:21.268084 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:21.268243 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:21.268269 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.268304 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:21.268334 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.268383 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.268472 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:21.251214 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.228425 27596 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.194841 27702 consensus_queue.cc:237] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.260999 27653 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:21.228120 27727 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:21.248430 27178 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.205826 26497 rpcz_store.cc:276] 0502 08:06:17.612337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:20.422644 (+2810307us) service_pool.cc:225] Handling call
0502 08:06:20.423170 (+ 526us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:20.423371 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 2
0502 08:06:20.893146 (+469775us) log.cc:842] Serialized 14 byte log entry
0502 08:06:21.085042 (+191896us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.093629 (+ 8587us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:21.120522 (+ 26893us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.155027 (+ 34505us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.169006 (+ 13979us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11400,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":8872,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":258399,"threads_started":2,"wal-append.queue_time_us":285913}
W20240502 08:06:21.264459 26646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55496 (request call id 454) took 2559 ms (client timeout 1828 ms). Trace:
I20240502 08:06:21.238770 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.245191 27542 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:21.270642 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 850) took 3577 ms. Trace:
I20240502 08:06:21.272172 27769 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:21.302110 26498 rpcz_store.cc:276] 0502 08:06:17.695416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.269776 (+3574360us) service_pool.cc:225] Handling call
0502 08:06:21.270105 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:21.270265 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:21.270291 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.270333 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:21.270388 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.270469 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.270584 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:21.337697 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:21.341991 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 145946us
I20240502 08:06:21.284708 27768 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:21.345034 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.291009 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 852) took 3451 ms. Trace:
I20240502 08:06:21.348448 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.290860 27727 raft_consensus.cc:686] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:19.095644 26467 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60752 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 916)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 894)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 913)
W20240502 08:06:21.287552 26646 rpcz_store.cc:269] 0502 08:06:18.689397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:18.956080 (+266683us) service_pool.cc:225] Handling call
0502 08:06:21.117787 (+2161707us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b5800204398. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f97011ba3b1 00007f97011ba262 00007f970a0ac889 00007f9708fe93c0 00007f97090af3bb 00007f97090af441 00007f970905ef60 00007f970905e844 00007f9702742c14 00007f9702742ba2
0502 08:06:21.248595 (+130808us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7926144}
I20240502 08:06:21.288118 27792 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.355903 27596 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:21.360935 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.284943 27746 raft_consensus.cc:686] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.365170 26498 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
W20240502 08:06:21.366101 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 123338us
W20240502 08:06:21.367141 26541 leader_election.cc:334] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.292s (SENT)
I20240502 08:06:21.342334 27680 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:21.372577 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 853) took 3370 ms. Trace:
I20240502 08:06:21.372869 26496 rpcz_store.cc:276] 0502 08:06:17.999443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.368729 (+3369286us) service_pool.cc:225] Handling call
0502 08:06:21.368930 (+ 201us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:21.369036 (+ 106us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:21.369049 (+ 13us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.369076 (+ 27us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:21.369091 (+ 15us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.369120 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.369180 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:06:21.371306 26615 leader_election.cc:334] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.795s (SENT)
I20240502 08:06:21.373889 26615 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:21.379668 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:21.379660 26615 leader_election.cc:334] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.818s (SENT)
I20240502 08:06:21.381237 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.381702 26615 leader_election.cc:334] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.970s (SENT)
I20240502 08:06:21.382311 26496 raft_consensus.cc:2396] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240502 08:06:21.382802 26615 leader_election.cc:334] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.001s (SENT)
I20240502 08:06:21.371448 26541 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.365592 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.384101 26615 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:21.386118 26542 leader_election.cc:334] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Timed out: RequestConsensusVote RPC to 127.24.194.3:40473 timed out after 1.544s (SENT)
W20240502 08:06:21.386920 26542 leader_election.cc:334] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Timed out: RequestConsensusVote RPC to 127.24.194.3:40473 timed out after 1.736s (SENT)
I20240502 08:06:21.386672 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:21.387398 26542 leader_election.cc:334] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Timed out: RequestConsensusVote RPC to 127.24.194.3:40473 timed out after 1.765s (SENT)
I20240502 08:06:21.388988 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:21.390077 26646 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:21.371874 26495 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:21.387687 26542 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.368639 27653 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5216c4a3a33f47ac88fa35704487f77e)
W20240502 08:06:21.395679 26541 leader_election.cc:334] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.493s (SENT)
I20240502 08:06:21.396540 26541 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.399194 27746 consensus_queue.cc:237] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:21.398298 26615 leader_election.cc:334] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.037s (SENT)
I20240502 08:06:21.401927 27653 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:21.402541 26542 leader_election.cc:334] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): Timed out: RequestConsensusVote RPC to 127.24.194.3:40473 timed out after 1.929s (SENT)
I20240502 08:06:21.407240 27692 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:21.383405 27750 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:21.400880 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.409569 26615 leader_election.cc:334] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.064s (SENT)
I20240502 08:06:21.411260 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.412501 27542 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:21.412756 27792 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:21.413323 27727 consensus_queue.cc:237] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.416090 26498 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.395527 27607 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.397970 27796 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:21.418213 26616 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.408450 27769 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.397617 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.400934 26494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 847) took 3752 ms. Trace:
I20240502 08:06:21.445648 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.446940 26615 leader_election.cc:334] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.108s (SENT)
I20240502 08:06:21.448211 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.365847 26499 rpcz_store.cc:276] 0502 08:06:17.839170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.271576 (+3432406us) service_pool.cc:225] Handling call
0502 08:06:21.271910 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:21.272085 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:21.272401 (+ 316us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.273313 (+ 912us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:21.281213 (+ 7900us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.284130 (+ 2917us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.290549 (+ 6419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:21.414492 27794 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.453505 26496 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
W20240502 08:06:21.457906 26541 leader_election.cc:334] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.858s (SENT)
I20240502 08:06:21.458427 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.459288 26496 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.419384 27542 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.456557 26616 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.462982 27750 consensus_queue.cc:1035] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:21.468894 27542 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.470407 27607 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.451728 26494 rpcz_store.cc:276] 0502 08:06:17.646429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.258518 (+3612089us) service_pool.cc:225] Handling call
0502 08:06:21.258966 (+ 448us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:21.259276 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 3
0502 08:06:21.259664 (+ 388us) log.cc:842] Serialized 14 byte log entry
0502 08:06:21.371310 (+111646us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0502 08:06:21.371830 (+ 520us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:21.378904 (+ 7074us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:21.385691 (+ 6787us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:21.398316 (+ 12625us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6781,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":1611,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":5615,"thread_start_us":105559,"threads_started":1,"wal-append.queue_time_us":143286}
I20240502 08:06:21.473819 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.474309 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.475148 26493 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.474426 26498 raft_consensus.cc:2362] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:21.478443 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.478724 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.479211 26500 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.479904 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.481192 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.481185 26541 leader_election.cc:334] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.050s (SENT)
I20240502 08:06:21.482133 26541 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.483204 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.486745 27794 raft_consensus.cc:2726] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.455881 27696 raft_consensus.cc:2781] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:21.487282 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.487911 26500 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:21.488689 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.489609 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.489217 26641 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:21.491313 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.492339 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.493049 26500 raft_consensus.cc:2379] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
W20240502 08:06:21.493352 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 45136us
I20240502 08:06:21.493782 26496 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.494608 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.495187 26496 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:21.495894 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.496676 26498 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:21.491724 27787 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:21.500695 26616 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.508378 26498 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:21.504753 26542 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.509563 26496 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:21.515170 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.523094 26500 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:21.503791 27696 raft_consensus.cc:686] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.519382 26616 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.513118 27514 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:21.525104 27680 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:21.525640 27514 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:21.526083 27514 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.525324 27696 consensus_queue.cc:237] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.526247 27689 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:21.527611 27689 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:21.532704 26542 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.535853 26616 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.539228 26542 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.541396 26542 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.542017 26496 raft_consensus.cc:3032] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:21.558635 26496 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:21.504050 27802 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:21.563390 26541 leader_election.cc:334] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.535s (SENT)
W20240502 08:06:21.564003 26541 leader_election.cc:334] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.544s (SENT)
I20240502 08:06:21.564036 26616 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.564443 26541 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.566956 27802 raft_consensus.cc:482] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:21.567469 27802 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.539095 27805 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.558249 27794 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:21.574663 27805 raft_consensus.cc:2741] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240502 08:06:21.575076 27794 raft_consensus.cc:2726] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:21.539283 27769 raft_consensus.cc:2781] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:21.560194 27803 raft_consensus.cc:2741] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240502 08:06:21.518308 27750 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20240502 08:06:21.576560 26399 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:21.603832 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 62513.3us
I20240502 08:06:21.604678 27769 raft_consensus.cc:686] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.562309 27727 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
W20240502 08:06:21.728156 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 164127us
I20240502 08:06:21.754860 26542 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:21.764783 26541 leader_election.cc:334] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.554s (SENT)
I20240502 08:06:21.563534 26496 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.765658 26541 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:21.756831 27144 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.763s (SENT) [suppressed 105 similar messages]
I20240502 08:06:21.753511 26399 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 2 to 4, pending config changed. New cstate: current_term: 4 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:21.765939 26496 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:21.766889 27805 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:21.767062 26495 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.767500 27805 raft_consensus.cc:686] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:21.768203 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.768438 27805 consensus_queue.cc:237] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.769467 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.593880 27702 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:21.782915 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.783596 26501 raft_consensus.cc:3037] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.787365 26500 raft_consensus.cc:1261] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:21.788306 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.788744 26500 raft_consensus.cc:2362] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:21.789323 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.791834 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.793732 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:21.794407 26541 leader_election.cc:334] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.736s (SENT)
I20240502 08:06:21.794909 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.795002 26541 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.795817 26500 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.798759 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:21.792238 26542 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.799644 26573 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:21.803856 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.804630 26500 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:21.805285 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.805725 26500 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:21.834025 26501 raft_consensus.cc:2454] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:21.784513 27727 raft_consensus.cc:686] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.902650 26542 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.903285 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.901412 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.932814 26573 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:21.932871 26500 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:21.792299 27514 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.791882 27702 raft_consensus.cc:686] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:21.944001 26573 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:21.802532 27794 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:21.946467 27794 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:21.946882 27794 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:21.947328 27794 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:21.769830 27802 raft_consensus.cc:504] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:21.951460 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.951969 27802 leader_election.cc:288] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:21.806461 27810 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:21.933844 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.932467 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60752 (request call id 891) took 1309 ms. Trace:
I20240502 08:06:21.957288 26501 rpcz_store.cc:276] 0502 08:06:20.623335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.782663 (+1159328us) service_pool.cc:225] Handling call
0502 08:06:21.861906 (+ 79243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:06:21.900490 27803 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:21.945416 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.932937 26542 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.962152 26542 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.931494 27542 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:21.954289 26541 leader_election.cc:334] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.828s (SENT)
I20240502 08:06:21.907084 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:21.957994 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.966339 26541 leader_election.cc:334] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.928s (SENT)
W20240502 08:06:21.966884 26541 leader_election.cc:334] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.929s (SENT)
I20240502 08:06:21.958873 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.953686 26569 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:21.964601 26616 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.972270 26569 raft_consensus.cc:1261] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:21.973615 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.963816 26573 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:21.975049 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.977094 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.968030 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:21.977331 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 220508us
I20240502 08:06:21.967123 26541 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:21.983052 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.984061 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:21.984674 26500 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:21.974354 26499 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:21.985307 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.985559 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.986621 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.987366 26500 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:21.986557 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:21.962949 27803 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:21.989102 26573 raft_consensus.cc:2454] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:22.036525 26616 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:21.968298 27810 raft_consensus.cc:2741] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240502 08:06:21.991134 27542 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:22.031351 27803 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:21.952760 27769 consensus_queue.cc:237] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.068203 27638 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:22.075459 27638 raft_consensus.cc:686] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:22.076704 27638 consensus_queue.cc:237] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.068401 27809 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:22.065320 27542 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:22.072610 27810 raft_consensus.cc:2726] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:22.111050 27809 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:22.116623 27809 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:22.124670 27809 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:22.155318 26541 leader_election.cc:334] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.021s (SENT)
I20240502 08:06:21.954896 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:22.158481 26541 leader_election.cc:334] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.082s (SENT)
I20240502 08:06:22.159230 26541 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:22.160300 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.164268 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.165479 26500 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:22.166056 26567 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:22.188639 27542 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:22.189429 27823 log.cc:924] Time spent T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20240502 08:06:22.190707 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.192420 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.193017 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.194943 26495 raft_consensus.cc:2396] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20240502 08:06:22.193264 26499 raft_consensus.cc:2454] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:22.201781 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.205227 26495 raft_consensus.cc:2379] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
W20240502 08:06:22.211032 26615 leader_election.cc:334] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.063s (SENT)
I20240502 08:06:22.211920 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.066022 27702 consensus_queue.cc:237] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.218673 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.210152 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.222692 27805 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:22.224141 26499 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.225018 26616 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:22.225435 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.226670 26499 raft_consensus.cc:2362] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:22.053052 27727 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.227133 27809 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:22.226761 27802 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:22.219664 27514 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:22.208142 27653 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:22.228461 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.238101 27752 consensus_queue.cc:1035] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:22.239388 27665 raft_consensus.cc:2781] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:22.212158 26569 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:22.267541 26615 leader_election.cc:334] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.124s (SENT)
I20240502 08:06:22.270334 27665 raft_consensus.cc:686] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
W20240502 08:06:22.270741 26641 consensus_meta.cc:287] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.769s user 0.014s sys 0.011s
I20240502 08:06:22.272112 27665 consensus_queue.cc:237] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.315021 26499 raft_consensus.cc:1261] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:22.321166 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.288691 27542 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.377076 26495 raft_consensus.cc:2396] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:22.329407 27620 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:22.317133 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 346805us
I20240502 08:06:22.397006 27803 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:22.374972 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 107529us
W20240502 08:06:22.374704 26541 leader_election.cc:334] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.151s (SENT)
W20240502 08:06:22.402014 26522 debug-util.cc:398] Leaking SignalData structure 0x7b0800359600 after lost signal to thread 26525
I20240502 08:06:22.402899 27794 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.407580 27542 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:22.416240 26542 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:22.417531 26570 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
W20240502 08:06:22.417593 26496 consensus_meta.cc:287] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.007s
I20240502 08:06:22.418761 26496 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.419481 27803 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:22.410171 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.414192 26541 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:22.402946 27751 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:22.421059 27809 raft_consensus.cc:2741] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240502 08:06:22.421969 27809 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:22.425094 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 54984.1us
W20240502 08:06:22.414686 26522 debug-util.cc:398] Leaking SignalData structure 0x7b0800359640 after lost signal to thread 26526
I20240502 08:06:22.417958 26542 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:22.427157 27809 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:22.428126 26542 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:22.429972 27796 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:22.430534 27796 raft_consensus.cc:686] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:22.431696 27796 consensus_queue.cc:237] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.442700 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.436139 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
W20240502 08:06:22.451438 26522 debug-util.cc:398] Leaking SignalData structure 0x7b080034b2a0 after lost signal to thread 26600
I20240502 08:06:22.442440 27751 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:22.454058 26500 raft_consensus.cc:2362] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:22.489526 26542 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:22.501883 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.502873 26495 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:22.505057 26522 debug-util.cc:398] Leaking SignalData structure 0x7b08001e1f80 after lost signal to thread 26674
W20240502 08:06:22.505548 26522 debug-util.cc:398] Leaking SignalData structure 0x7b08001eeb60 after lost signal to thread 26675
W20240502 08:06:22.507243 26522 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc960 after lost signal to thread 27204
I20240502 08:06:22.514580 27833 raft_consensus.cc:2781] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:22.515722 27833 raft_consensus.cc:686] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
W20240502 08:06:22.516219 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 78827.4us
I20240502 08:06:22.517290 26542 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:22.508652 26541 leader_election.cc:334] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.170s (SENT)
I20240502 08:06:22.517143 27833 consensus_queue.cc:237] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.524114 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.496850 27794 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:22.524581 26396 catalog_manager.cc:5484] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:22.549685 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.555754 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.520473 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.565449 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.564070 26542 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:22.569106 27840 log.cc:924] Time spent T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240502 08:06:22.568923 27832 raft_consensus.cc:2781] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:22.589764 26501 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:22.594702 26398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39264 (request call id 129) took 8121 ms. Trace:
I20240502 08:06:22.602188 26500 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.615171 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.597353 27832 raft_consensus.cc:686] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:22.622643 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.623679 26576 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:22.626614 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.630008 26399 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52756 (request call id 22) took 1217 ms. Trace:
I20240502 08:06:22.630589 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.631325 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.633078 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.631429 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.617666 27833 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:22.657003 26501 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:22.633924 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.664474 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.665199 26571 raft_consensus.cc:1261] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.666369 27609 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:22.667021 27609 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.667057 27833 raft_consensus.cc:686] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:22.669519 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:22.669903 27850 log.cc:924] Time spent T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.061s user 0.000s sys 0.004s
I20240502 08:06:22.670074 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.671301 27794 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:22.671547 27833 consensus_queue.cc:237] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.671983 27794 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.674351 27794 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:22.674548 27832 consensus_queue.cc:237] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.683439 26570 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.671958 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.662348 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.686590 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.687367 26571 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:22.601758 26398 rpcz_store.cc:276] 0502 08:06:14.442849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:14.443083 (+ 234us) service_pool.cc:225] Handling call
0502 08:06:14.619439 (+176356us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54000037d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a1d30 00007f970412517a 00007f970a076b1c 00007f970a170a8e 00007f9708484a44 00007f97084849d2 00007f9708484901 00007f970848487c 00007f9708483b32 00007f9702543bcc 00007f9702543006
0502 08:06:22.562079 (+7942640us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:21.272031 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:21.383911 (+111880us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:21.386377 (+ 2466us) write_op.cc:427] Acquired schema lock
0502 08:06:21.394410 (+ 8033us) tablet.cc:660] Decoding operations
0502 08:06:21.475628 (+ 81218us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:21.475709 (+ 81us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:21.475739 (+ 30us) tablet.cc:683] Acquiring locks for 10 operations
0502 08:06:21.476503 (+ 764us) tablet.cc:699] Row locks acquired
0502 08:06:21.476521 (+ 18us) write_op.cc:260] PREPARE: finished
0502 08:06:21.476664 (+ 143us) write_op.cc:270] Start()
0502 08:06:21.476776 (+ 112us) write_op.cc:276] Timestamp: P: 1714637181476622 usec, L: 0
0502 08:06:21.476799 (+ 23us) op_driver.cc:331] REPLICATION: starting
0502 08:06:21.477391 (+ 592us) log.cc:842] Serialized 4003 byte log entry
0502 08:06:22.125886 (+648495us) op_driver.cc:444] REPLICATION: finished
0502 08:06:22.179761 (+ 53875us) write_op.cc:301] APPLY: starting
0502 08:06:22.199602 (+ 19841us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:22.200032 (+ 430us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:22.200265 (+ 233us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:22.367290 (+167025us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:22.367882 (+ 592us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0502 08:06:22.368005 (+ 123us) write_op.cc:312] APPLY: finished
0502 08:06:22.368762 (+ 757us) log.cc:842] Serialized 76 byte log entry
0502 08:06:22.390544 (+ 21782us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.391026 (+ 482us) write_op.cc:446] Released schema lock
0502 08:06:22.391126 (+ 100us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:22.396820 (+ 5694us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.396857 (+ 37us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2850,"reported_tablets":11,"spinlock_wait_cycles":397058816,"child_traces":[["op",{"apply.queue_time_us":1483,"apply.run_cpu_time_us":9508,"apply.run_wall_time_us":269653,"num_ops":10,"prepare.queue_time_us":236813,"prepare.run_cpu_time_us":14869,"prepare.run_wall_time_us":261782,"raft.queue_time_us":471993,"raft.run_cpu_time_us":15356,"raft.run_wall_time_us":158587,"replication_time_us":648989,"thread_start_us":602936,"threads_started":4,"wal-append.queue_time_us":145320}]]}
I20240502 08:06:22.680606 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.691099 26496 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.693377 26497 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.630504 26399 rpcz_store.cc:276] 0502 08:06:21.412554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.413542 (+ 988us) service_pool.cc:225] Handling call
0502 08:06:22.629756 (+1216214us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:21.950226 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:21.950709 (+ 483us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:21.950750 (+ 41us) write_op.cc:427] Acquired schema lock
0502 08:06:21.950768 (+ 18us) tablet.cc:660] Decoding operations
0502 08:06:21.951294 (+ 526us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:21.951340 (+ 46us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:21.951365 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:06:21.951620 (+ 255us) tablet.cc:699] Row locks acquired
0502 08:06:21.951639 (+ 19us) write_op.cc:260] PREPARE: finished
0502 08:06:21.951773 (+ 134us) write_op.cc:270] Start()
0502 08:06:21.951880 (+ 107us) write_op.cc:276] Timestamp: P: 1714637181951727 usec, L: 0
0502 08:06:21.951899 (+ 19us) op_driver.cc:331] REPLICATION: starting
0502 08:06:22.031496 (+ 79597us) log.cc:842] Serialized 1283 byte log entry
0502 08:06:22.189510 (+158014us) op_driver.cc:444] REPLICATION: finished
0502 08:06:22.397596 (+208086us) write_op.cc:301] APPLY: starting
0502 08:06:22.397713 (+ 117us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:22.397868 (+ 155us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:22.410579 (+ 12711us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:22.456797 (+ 46218us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:22.456973 (+ 176us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:06:22.457019 (+ 46us) write_op.cc:312] APPLY: finished
0502 08:06:22.457259 (+ 240us) log.cc:842] Serialized 28 byte log entry
0502 08:06:22.457487 (+ 228us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.488158 (+ 30671us) write_op.cc:446] Released schema lock
0502 08:06:22.490354 (+ 2196us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:22.495782 (+ 5428us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.497017 (+ 1235us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":56,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":207732,"apply.run_cpu_time_us":6022,"apply.run_wall_time_us":120566,"num_ops":2,"prepare.queue_time_us":1680,"prepare.run_cpu_time_us":3169,"prepare.run_wall_time_us":103992,"replication_time_us":237382,"thread_start_us":83446,"threads_started":1}]]}
I20240502 08:06:22.694900 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.697582 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.699085 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.711998 26650 raft_consensus.cc:2379] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:22.687377 27609 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:22.693681 27794 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:22.729400 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.730595 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.733004 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.733698 26567 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:22.735380 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.735080 27794 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:22.736191 26646 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:22.737613 27794 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:22.746006 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.744791 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.751571 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.752701 26396 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39274 (request call id 138) took 9743 ms. Trace:
I20240502 08:06:22.754029 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.761478 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.766331 26576 raft_consensus.cc:2454] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:22.767557 26496 raft_consensus.cc:2379] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.792304 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.793484 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.794693 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.767025 26395 catalog_manager.cc:5484] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:22.795070 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.795982 26650 raft_consensus.cc:2379] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:22.796911 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.796595 26395 catalog_manager.cc:5484] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:22.768875 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.798190 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.798997 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.799343 26650 raft_consensus.cc:2379] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:22.799858 26501 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:22.800930 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.769719 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.802155 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.802837 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.803246 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.803987 26576 raft_consensus.cc:2379] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:22.803927 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.804843 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.805083 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.805457 26576 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:22.805771 26498 raft_consensus.cc:2379] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.806309 26499 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:22.770570 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.771421 26492 raft_consensus.cc:2379] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:22.753075 26396 rpcz_store.cc:276] 0502 08:06:13.006760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:13.006930 (+ 170us) service_pool.cc:225] Handling call
0502 08:06:22.750487 (+9743557us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:22.723389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:22.723946 (+ 557us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:22.724004 (+ 58us) write_op.cc:427] Acquired schema lock
0502 08:06:22.724016 (+ 12us) tablet.cc:660] Decoding operations
0502 08:06:22.724314 (+ 298us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:22.724400 (+ 86us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:22.724423 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:06:22.724686 (+ 263us) tablet.cc:699] Row locks acquired
0502 08:06:22.724698 (+ 12us) write_op.cc:260] PREPARE: finished
0502 08:06:22.724839 (+ 141us) write_op.cc:270] Start()
0502 08:06:22.724961 (+ 122us) write_op.cc:276] Timestamp: P: 1714637182724777 usec, L: 0
0502 08:06:22.724975 (+ 14us) op_driver.cc:331] REPLICATION: starting
0502 08:06:22.725587 (+ 612us) log.cc:842] Serialized 533 byte log entry
0502 08:06:22.728822 (+ 3235us) op_driver.cc:444] REPLICATION: finished
0502 08:06:22.733130 (+ 4308us) write_op.cc:301] APPLY: starting
0502 08:06:22.733218 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:22.733325 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:22.733437 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:22.733643 (+ 206us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:22.733783 (+ 140us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:06:22.733826 (+ 43us) write_op.cc:312] APPLY: finished
0502 08:06:22.734007 (+ 181us) log.cc:842] Serialized 22 byte log entry
0502 08:06:22.734184 (+ 177us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.734261 (+ 77us) write_op.cc:446] Released schema lock
0502 08:06:22.734306 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:22.734543 (+ 237us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:22.734568 (+ 25us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":35488,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":1233,"apply.run_cpu_time_us":1646,"apply.run_wall_time_us":1646,"num_ops":1,"prepare.queue_time_us":94887,"prepare.run_cpu_time_us":2479,"prepare.run_wall_time_us":2487,"replication_time_us":2715,"thread_start_us":31450,"threads_started":1}]]}
I20240502 08:06:22.771682 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.772856 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.778349 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:22.778921 27853 log.cc:924] Time spent T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20240502 08:06:22.779676 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.781318 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.782186 26569 raft_consensus.cc:1226] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:22.783063 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5145fc92dff54e38b92911a9a14b978e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.784241 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.784663 26645 raft_consensus.cc:1261] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:22.786933 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.787829 26573 raft_consensus.cc:1226] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:22.787895 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.787871 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.788762 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.788972 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.789515 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.790247 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.806453 26574 raft_consensus.cc:1226] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:22.797839 26571 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:22.770730 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:22.855091 26615 leader_election.cc:334] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.571s (SENT)
I20240502 08:06:22.914052 26497 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:22.855330 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.807956 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.846575 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.834390 26395 catalog_manager.cc:5484] T 12b2f57a1a34420e98d2d4d7f2f9162e P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:22.917382 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.918530 26500 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 2.
I20240502 08:06:22.919169 26501 raft_consensus.cc:2379] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
W20240502 08:06:22.865603 27859 log.cc:924] Time spent T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240502 08:06:22.920527 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.923089 26497 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.923372 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.921197 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.925369 26542 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:22.919493 26649 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:22.926684 26576 raft_consensus.cc:2454] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:22.927366 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.928268 26575 raft_consensus.cc:1226] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:22.929847 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.930493 26500 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 4: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.931403 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.930780 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.933213 26496 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:22.933971 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.934952 26569 raft_consensus.cc:2379] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:22.937248 26571 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.937881 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.938925 27833 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:22.941434 27833 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8f9fe1f3763547388c4301b13a127aaa)
I20240502 08:06:22.938650 26650 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:22.942416 26498 raft_consensus.cc:3037] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:22.919647 26494 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:22.943215 26568 raft_consensus.cc:1226] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:22.943396 26575 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.941830 27833 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:22.946537 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 105936us
I20240502 08:06:22.947806 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.960245 26500 raft_consensus.cc:2362] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3. Re-sending same reply.
I20240502 08:06:22.961345 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:22.961853 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.962781 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73e56c17a6f4770a16b443ab75ea198" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:22.962262 26615 leader_election.cc:334] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.578s (SENT)
I20240502 08:06:22.969640 27794 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240502 08:06:22.973822 26542 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:22.979103 27832 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:22.964426 26395 catalog_manager.cc:5484] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:22.980427 27832 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:22.981701 26496 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.982340 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.982216 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:22.983376 26575 raft_consensus.cc:2379] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 in current term 5: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:22.983851 27832 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:22.984442 26499 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:22.988198 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.990625 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.991253 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.992022 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.992494 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.993415 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:22.993908 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:22.994699 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b2256f7eb5420792d8042fb8e30d0b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.995229 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.995704 26574 raft_consensus.cc:1226] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:22.995841 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:22.989074 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:22.996286 26567 raft_consensus.cc:2379] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 4: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:22.994202 26570 raft_consensus.cc:2354] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4.
I20240502 08:06:22.998584 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.000643 26499 raft_consensus.cc:3032] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:23.001015 26499 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.001376 26492 raft_consensus.cc:3032] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:23.001792 26492 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.001857 26499 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.003082 26499 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.002990 26492 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.004238 26492 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:23.026644 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:23.026993 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82da19e5ed7d415588714d971127c197" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.027977 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.031246 26642 raft_consensus.cc:1226] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.032393 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:23.032747 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 72790.7us
I20240502 08:06:23.026379 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:23.034595 26572 raft_consensus.cc:1226] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.034736 26644 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240502 08:06:23.033478 26615 leader_election.cc:334] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.621s (SENT)
I20240502 08:06:23.035497 26615 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.035733 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.037031 26567 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.037142 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.038475 26567 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.038786 26501 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.038735 27609 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:23.040264 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.040704 26541 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:23.041177 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.041558 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:23.043956 26615 leader_election.cc:334] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.654s (SENT)
I20240502 08:06:23.044744 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.044860 27794 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:23.046805 26575 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.060832 26645 raft_consensus.cc:2354] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:23.046589 26395 catalog_manager.cc:5484] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.061229 26499 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.063004 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.047621 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.048121 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.048381 26570 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240502 08:06:23.048825 26493 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:23.049229 26573 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.051667 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.052079 27750 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.066756 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.067193 26645 raft_consensus.cc:1261] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:23.069578 26570 raft_consensus.cc:1226] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.073570 27832 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.074807 27747 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.076165 27832 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:23.065274 26493 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.052382 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:23.078363 26493 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.079689 26493 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.082455 26569 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.062151 26395 catalog_manager.cc:5484] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.084519 27833 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.049892 26574 raft_consensus.cc:1226] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.057189 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:23.086462 26495 raft_consensus.cc:3032] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:23.086831 26495 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.087034 26570 raft_consensus.cc:1226] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.087286 27796 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:23.088246 26642 raft_consensus.cc:1226] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.058133 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.059165 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:23.053923 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.060103 26644 raft_consensus.cc:1226] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.062932 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.047520 27609 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:23.065799 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.092365 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.082307 27824 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20240502 08:06:23.094215 26500 raft_consensus.cc:3032] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:23.094578 26500 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.094676 26501 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:23.087723 26495 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.095503 26500 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.096702 26642 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.097103 26645 raft_consensus.cc:1226] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: [2.2-4.3]
W20240502 08:06:23.082275 26615 leader_election.cc:334] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.752s (SENT)
I20240502 08:06:23.098546 26499 raft_consensus.cc:3032] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:23.098903 26499 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.099001 26642 raft_consensus.cc:1226] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.059391 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.087340 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.056057 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.084250 26395 catalog_manager.cc:5484] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.099653 26647 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.104693 26498 raft_consensus.cc:2454] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:23.097028 26495 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:23.106843 26541 leader_election.cc:334] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.517s (SENT)
I20240502 08:06:23.109596 26541 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.113499 26572 raft_consensus.cc:2354] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:23.114764 26496 raft_consensus.cc:2396] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240502 08:06:23.116329 26500 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:23.118834 26641 raft_consensus.cc:1226] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.110661 26395 catalog_manager.cc:5484] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
W20240502 08:06:23.127007 26541 leader_election.cc:334] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.725s (SENT)
I20240502 08:06:23.114547 27832 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:23.127643 26541 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.125355 26616 leader_election.cc:302] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.129043 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.129666 26642 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.132908 26499 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.144196 26573 raft_consensus.cc:1226] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.143678 26642 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:23.147794 26542 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.148250 26499 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.148427 26395 catalog_manager.cc:5484] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.135178 27796 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:23.126500 27833 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:23.154656 26645 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:23.156263 26616 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.153005 27809 raft_consensus.cc:2781] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:23.153297 27824 raft_consensus.cc:2781] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:23.160568 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9fcbac5f24ae8b0c8f071ae34229f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.161307 26616 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.152516 26395 catalog_manager.cc:5484] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.161532 27751 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:23.176872 27824 raft_consensus.cc:686] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:23.176884 27747 raft_consensus.cc:2781] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:23.177846 27824 consensus_queue.cc:237] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.178798 27747 raft_consensus.cc:686] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:23.180052 27747 consensus_queue.cc:237] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.163058 27809 raft_consensus.cc:686] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:23.184799 27809 consensus_queue.cc:237] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.164827 26492 raft_consensus.cc:1261] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.164208 26641 raft_consensus.cc:1261] T 1d753d3fcb9249c28f415af2ab615999 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.176066 26395 catalog_manager.cc:5484] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 2 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.190522 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.159576 27833 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240502 08:06:23.188678 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.192914 26395 catalog_manager.cc:5484] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:23.181898 27751 raft_consensus.cc:686] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:23.194624 27750 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240502 08:06:23.218528 26645 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.219941 27751 consensus_queue.cc:237] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.224925 26499 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:06:23.242357 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 48094.8us
I20240502 08:06:23.241839 26500 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.244277 27542 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240502 08:06:23.243379 26649 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:23.265373 26495 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:23.270907 26615 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.271803 27750 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:23.273008 26647 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:23.281258 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6f8df55d8c4456ac8d84fdcb919e55" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.276945 27620 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:23.291133 27796 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:23.292830 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.294353 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.294555 26641 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.296051 26646 raft_consensus.cc:1226] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.298734 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.298988 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.284838 27542 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240502 08:06:23.300418 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.302031 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.302484 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.302829 26571 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
W20240502 08:06:23.303373 27868 log.cc:924] Time spent T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240502 08:06:23.303866 27835 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:23.304389 27835 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:23.307065 26615 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.302511 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.308620 26649 raft_consensus.cc:2379] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 4: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:23.309139 27835 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:23.309592 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.311661 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.313396 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.314858 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:23.316213 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.320174 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.320917 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:23.322271 26645 raft_consensus.cc:1226] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.322389 27145 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.323621 27835 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:23.324672 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e067615c3a4148d597769a9f8a757e38" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:23.334226 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 46225.5us
I20240502 08:06:23.320344 26493 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240502 08:06:23.336686 26642 raft_consensus.cc:1226] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:23.336395 27832 consensus_queue.cc:1035] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20240502 08:06:23.339713 27144 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.341667 27835 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:23.342208 26542 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.343180 26542 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.344359 27794 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:23.344995 27794 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:23.346065 27794 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.344167 27875 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.347489 27809 raft_consensus.cc:2781] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:23.335225 27747 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:23.348086 27809 raft_consensus.cc:686] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:23.349371 27809 consensus_queue.cc:237] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.348117 27747 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:23.351290 27747 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
W20240502 08:06:23.339113 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62030.5us
I20240502 08:06:23.335196 26493 pending_rounds.cc:76] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Aborting all ops after (but not including) 2
I20240502 08:06:23.361457 26492 raft_consensus.cc:3032] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:23.363435 26492 raft_consensus.cc:729] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.363539 26493 pending_rounds.cc:97] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240502 08:06:23.363952 26493 raft_consensus.cc:2866] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:23.364460 26492 consensus_queue.cc:260] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.365671 26492 raft_consensus.cc:3037] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.361487 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:23.366528 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.366724 26495 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:23.367350 26641 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:23.362522 27835 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:23.368721 27881 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
W20240502 08:06:23.373539 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 66581.2us
I20240502 08:06:23.380818 26495 raft_consensus.cc:3032] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:23.381239 26495 raft_consensus.cc:729] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
W20240502 08:06:23.380038 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 54181.6us
I20240502 08:06:23.388916 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:23.392287 27824 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:23.392560 27723 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.421214 26615 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.406247 27751 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240502 08:06:23.428026 27609 consensus_queue.cc:1035] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.429435 27178 raft_consensus.cc:1226] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:23.439570 26573 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.452363 27845 consensus_queue.cc:1046] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20240502 08:06:23.444597 27542 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:23.415464 26495 consensus_queue.cc:260] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.459218 27542 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
W20240502 08:06:23.471791 26541 leader_election.cc:334] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.647s (SENT)
I20240502 08:06:23.472445 26541 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.468024 27542 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.482079 27723 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:23.468129 27620 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240502 08:06:23.485258 27751 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:23.480538 26495 raft_consensus.cc:3037] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.468423 27887 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240502 08:06:23.498431 27609 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240502 08:06:23.500283 27884 log.cc:924] Time spent T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.051s user 0.002s sys 0.000s
I20240502 08:06:23.500856 26492 raft_consensus.cc:1261] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
W20240502 08:06:23.503805 26541 leader_election.cc:334] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.681s (SENT)
I20240502 08:06:23.506870 26541 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.507877 27794 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:23.508291 26541 leader_election.cc:334] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.804s (SENT)
I20240502 08:06:23.508836 26541 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.509737 27794 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:23.510180 26541 leader_election.cc:334] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.916s (SENT)
I20240502 08:06:23.510739 26541 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.511590 27794 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:06:23.512019 26541 leader_election.cc:334] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.929s (SENT)
W20240502 08:06:23.513074 26541 leader_election.cc:334] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.517s (SENT)
W20240502 08:06:23.514010 26541 leader_election.cc:334] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.570s (SENT)
I20240502 08:06:23.514480 26541 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.515339 27794 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:23.504772 27747 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:23.495427 27809 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:23.519309 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.516511 26492 pending_rounds.cc:76] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Aborting all ops after (but not including) 1
W20240502 08:06:23.530223 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62329.7us
I20240502 08:06:23.530928 26571 raft_consensus.cc:1261] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.521981 26492 pending_rounds.cc:97] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240502 08:06:23.535061 26574 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:23.536852 27609 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240502 08:06:23.578495 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 73476.7us
I20240502 08:06:23.579300 26615 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.560454 26492 raft_consensus.cc:2866] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:23.585839 26493 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:23.589004 26492 pending_rounds.cc:97] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240502 08:06:23.599079 26492 raft_consensus.cc:2866] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:23.599961 27747 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:23.612160 26615 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.613189 27891 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:23.601291 27620 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:23.626253 26541 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.626178 27895 consensus_queue.cc:1035] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
W20240502 08:06:23.629809 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 57842.2us
I20240502 08:06:23.627185 27620 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:23.655014 27845 raft_consensus.cc:482] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:23.655601 27845 raft_consensus.cc:504] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.657481 27542 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.659552 27542 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:23.662616 27845 leader_election.cc:288] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:23.635562 27875 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:06:23.678671 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 52556.8us
I20240502 08:06:23.665467 27890 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:23.666657 27620 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:23.700464 26541 leader_election.cc:334] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.979s (SENT)
W20240502 08:06:23.705497 26541 leader_election.cc:334] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.022s (SENT)
W20240502 08:06:23.706764 26541 leader_election.cc:334] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.046s (SENT)
I20240502 08:06:23.707129 26615 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:23.707191 26541 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.708503 26644 raft_consensus.cc:1226] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.708739 26615 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.709668 27889 raft_consensus.cc:3032] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:23.710147 27889 raft_consensus.cc:729] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.711198 27889 consensus_queue.cc:260] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:23.712743 27889 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:23.699865 27890 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.715978 27890 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:23.718178 27609 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:23.718629 27609 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5216c4a3a33f47ac88fa35704487f77e)
I20240502 08:06:23.719022 27609 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.718212 27833 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:23.723409 26541 leader_election.cc:334] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.215s (SENT)
I20240502 08:06:23.724018 26541 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.725064 27833 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:23.724155 27723 raft_consensus.cc:2726] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:23.739451 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.745471 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:23.744976 26541 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
W20240502 08:06:23.751108 26467 connection.cc:232] RPC call timeout handler was delayed by -6.023026838000078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
W20240502 08:06:23.751657 26467 connection.cc:232] RPC call timeout handler was delayed by -5.929426838000154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20240502 08:06:23.751777 27870 raft_consensus.cc:2781] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
W20240502 08:06:23.752269 26467 connection.cc:232] RPC call timeout handler was delayed by -5.8754268380000667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
I20240502 08:06:23.752405 27870 raft_consensus.cc:686] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
W20240502 08:06:23.752701 26467 connection.cc:232] RPC call timeout handler was delayed by -5.84032683800001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20240502 08:06:23.753185 26467 connection.cc:232] RPC call timeout handler was delayed by -5.7152268380000351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
W20240502 08:06:23.753558 26467 connection.cc:232] RPC call timeout handler was delayed by -5.5847268380000514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20240502 08:06:23.754158 26467 connection.cc:232] RPC call timeout handler was delayed by -5.5694268380000267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20240502 08:06:23.754583 26467 connection.cc:232] RPC call timeout handler was delayed by -5.4378729270001713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
W20240502 08:06:23.754904 26467 connection.cc:232] RPC call timeout handler was delayed by -5.4378729270001713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
W20240502 08:06:23.756100 26467 connection.cc:232] RPC call timeout handler was delayed by -5.3559729270000389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20240502 08:06:23.781109 26467 connection.cc:232] RPC call timeout handler was delayed by -5.3460729270000229s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
W20240502 08:06:23.793515 26467 connection.cc:232] RPC call timeout handler was delayed by -5.3235729270002139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20240502 08:06:23.794090 26467 connection.cc:232] RPC call timeout handler was delayed by -5.3091729270001906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20240502 08:06:23.798213 26467 connection.cc:232] RPC call timeout handler was delayed by -5.1093729270000949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20240502 08:06:23.798609 26467 connection.cc:232] RPC call timeout handler was delayed by -4.9806729270001142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20240502 08:06:23.798991 26467 connection.cc:232] RPC call timeout handler was delayed by -4.9104729270000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20240502 08:06:23.799396 26467 connection.cc:232] RPC call timeout handler was delayed by -4.2039729270002226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28940000000000005s for a response
W20240502 08:06:23.800297 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 55394.5us
I20240502 08:06:23.749212 26495 raft_consensus.cc:1261] T 39e50c6374394235996663b59b4be48a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.803998 26650 raft_consensus.cc:1261] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:23.745478 27796 consensus_queue.cc:1046] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20240502 08:06:23.808338 27875 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:23.816470 26541 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.753515 27870 consensus_queue.cc:237] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:23.827590 27898 log.cc:924] Time spent T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20240502 08:06:23.837229 27609 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:23.844141 27890 consensus_queue.cc:1046] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:23.844700 27609 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:23.855542 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:23.845139 27890 raft_consensus.cc:3032] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
W20240502 08:06:23.866716 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 45065us
I20240502 08:06:23.862411 27845 raft_consensus.cc:2741] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240502 08:06:23.862368 27890 raft_consensus.cc:729] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:23.870712 27890 consensus_queue.cc:260] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.872217 27890 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:23.884927 26615 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.885877 27723 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:23.886322 26615 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.886334 27723 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:23.887079 27723 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.887749 26615 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:23.888258 27864 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.244s user 0.006s sys 0.000s
I20240502 08:06:23.890208 27824 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:06:23.900903 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 84484.9us
I20240502 08:06:23.901866 26541 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:23.898708 27887 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240502 08:06:23.907930 27908 log.cc:924] Time spent T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20240502 08:06:23.908967 26615 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.909917 27895 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:23.910188 27876 raft_consensus.cc:3032] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:23.910558 27876 raft_consensus.cc:729] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:23.911355 27876 consensus_queue.cc:260] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:23.912753 27876 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.913848 27895 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:06:23.916752 27911 log.cc:924] Time spent T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.077s user 0.002s sys 0.000s
I20240502 08:06:23.933149 26467 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:23.933144 27887 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 08:06:23.905205 27723 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.947438 27913 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:23.947958 27913 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:23.948343 27913 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:23.942867 27833 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:23.947511 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 45737.4us
I20240502 08:06:23.952934 26541 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.952437 27723 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:23.955657 26399 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 2 to 3, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1), VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2), VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3), VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4). New cstate: current_term: 4 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:23.919798 27620 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:23.976989 26650 raft_consensus.cc:1226] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:23.982792 27620 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:23.984141 26650 raft_consensus.cc:1226] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Rejecting Update request from peer 5216c4a3a33f47ac88fa35704487f77e for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:23.984216 26541 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:23.985631 26395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39272 (request call id 153) took 2606 ms. Trace:
W20240502 08:06:23.985564 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 61451.3us
I20240502 08:06:23.973822 27889 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:23.986835 27889 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:23.987370 27889 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:23.985991 27881 consensus_queue.cc:1046] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240502 08:06:23.987478 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:23.960965 27900 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:23.996397 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 62594.9us
I20240502 08:06:23.997820 27875 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:23.998982 27845 raft_consensus.cc:2726] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:23.989051 26395 rpcz_store.cc:276] 0502 08:06:21.392037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:21.392225 (+ 188us) service_pool.cc:225] Handling call
0502 08:06:22.242459 (+850234us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b54000037d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a1d30 00007f970412517a 00007f970a076b1c 00007f970a170a8e 00007f9708484a44 00007f97084849d2 00007f9708484901 00007f970848487c 00007f9708483b32 00007f9702543bcc 00007f9702543006
0502 08:06:23.985551 (+1743092us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:23.222743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:23.223233 (+ 490us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:23.223284 (+ 51us) write_op.cc:427] Acquired schema lock
0502 08:06:23.223313 (+ 29us) tablet.cc:660] Decoding operations
0502 08:06:23.224457 (+ 1144us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:23.224525 (+ 68us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:23.224553 (+ 28us) tablet.cc:683] Acquiring locks for 12 operations
0502 08:06:23.246285 (+ 21732us) tablet.cc:699] Row locks acquired
0502 08:06:23.247032 (+ 747us) write_op.cc:260] PREPARE: finished
0502 08:06:23.279834 (+ 32802us) write_op.cc:270] Start()
0502 08:06:23.282608 (+ 2774us) write_op.cc:276] Timestamp: P: 1714637183259925 usec, L: 0
0502 08:06:23.284313 (+ 1705us) op_driver.cc:331] REPLICATION: starting
0502 08:06:23.304689 (+ 20376us) log.cc:842] Serialized 4773 byte log entry
0502 08:06:23.439177 (+134488us) op_driver.cc:444] REPLICATION: finished
0502 08:06:23.594150 (+154973us) write_op.cc:301] APPLY: starting
0502 08:06:23.597552 (+ 3402us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:23.638693 (+ 41141us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:23.666525 (+ 27832us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:23.718934 (+ 52409us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:23.719234 (+ 300us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0502 08:06:23.719285 (+ 51us) write_op.cc:312] APPLY: finished
0502 08:06:23.719649 (+ 364us) log.cc:842] Serialized 88 byte log entry
0502 08:06:23.719818 (+ 169us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:23.719965 (+ 147us) write_op.cc:446] Released schema lock
0502 08:06:23.720025 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:23.720302 (+ 277us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:23.720336 (+ 34us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1125,"reported_tablets":28,"spinlock_wait_cycles":123063040,"child_traces":[["op",{"apply.queue_time_us":135503,"apply.run_cpu_time_us":4399,"apply.run_wall_time_us":139538,"num_ops":12,"prepare.queue_time_us":1741,"prepare.run_cpu_time_us":5863,"prepare.run_wall_time_us":84787,"replication_time_us":150260,"thread_start_us":66687,"threads_started":1}]]}
W20240502 08:06:24.013756 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 69608.8us
I20240502 08:06:24.018604 26641 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:24.019637 27875 consensus_queue.cc:1046] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.020514 27875 raft_consensus.cc:3032] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:24.020814 27875 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:24.022763 26649 raft_consensus.cc:1226] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:24.021545 27875 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.024200 27875 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:24.024315 26650 raft_consensus.cc:1226] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:24.024705 27723 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240502 08:06:24.025120 27620 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240502 08:06:24.032444 26467 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:24.036438 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 52368.7us
I20240502 08:06:24.039616 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:24.042366 26541 leader_election.cc:334] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.716s (SENT)
I20240502 08:06:24.043087 26541 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:24.044786 27914 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:24.050231 26571 raft_consensus.cc:1261] T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:24.049167 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:24.038535 27890 raft_consensus.cc:2726] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:24.035189 27881 raft_consensus.cc:3032] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:24.033067 27870 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20240502 08:06:24.042363 27887 raft_consensus.cc:2726] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:24.058998 26575 raft_consensus.cc:1261] T 82da19e5ed7d415588714d971127c197 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:24.041047 27832 consensus_queue.cc:1046] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240502 08:06:24.062701 27832 raft_consensus.cc:3032] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:24.063052 27832 raft_consensus.cc:729] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:24.063861 27832 consensus_queue.cc:260] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.065055 27832 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:24.038292 27809 consensus_queue.cc:1046] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240502 08:06:24.066488 27809 raft_consensus.cc:3032] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:24.067792 27809 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:24.073439 27809 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.071942 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:24.076423 26644 raft_consensus.cc:1261] T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:24.059270 27881 raft_consensus.cc:729] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:24.081037 27809 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:24.067855 27913 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.112501 26467 leader_election.cc:398] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:24.101616 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:24.081712 27870 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
W20240502 08:06:24.135509 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 44592.9us
I20240502 08:06:24.140065 27609 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:24.169186 27751 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
W20240502 08:06:24.186803 27930 log.cc:924] Time spent T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240502 08:06:24.202517 27895 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.202682 27897 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240502 08:06:24.203418 26567 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:24.203965 27895 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.204212 27897 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.207072 26616 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:24.207846 27892 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.209224 27892 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:24.209697 27892 raft_consensus.cc:686] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:24.213440 26574 raft_consensus.cc:1226] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:24.218742 27892 consensus_queue.cc:237] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.220701 27913 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:24.237537 26541 leader_election.cc:334] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.003s (SENT)
I20240502 08:06:24.124435 27881 consensus_queue.cc:260] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:24.250608 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:24.286175 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 87785.5us
W20240502 08:06:24.284814 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 50302.9us
I20240502 08:06:24.326301 27891 consensus_queue.cc:1035] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:24.339946 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:24.340941 26493 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:24.342280 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.346462 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.347138 26492 raft_consensus.cc:2362] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4. Re-sending same reply.
I20240502 08:06:24.337592 27933 consensus_queue.cc:1046] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20240502 08:06:24.369560 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
W20240502 08:06:24.373065 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 69565.1us
I20240502 08:06:24.371510 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240502 08:06:24.374920 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 52410us
I20240502 08:06:24.379653 27723 raft_consensus.cc:3032] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:24.380287 27723 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
W20240502 08:06:24.381904 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 101711us
I20240502 08:06:24.381480 27723 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.382885 27723 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:24.384193 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.384783 26501 raft_consensus.cc:3037] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:24.277639 27889 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.297993 27542 consensus_queue.cc:1035] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
W20240502 08:06:24.483163 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 106160us
I20240502 08:06:24.376036 27881 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:24.487700 26500 raft_consensus.cc:1226] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:24.489012 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:24.490875 27889 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:24.494725 27937 log.cc:924] Time spent T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
W20240502 08:06:24.495790 27938 log.cc:924] Time spent T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20240502 08:06:24.500893 26541 leader_election.cc:334] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.019s (SENT)
I20240502 08:06:24.524052 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.525401 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.527606 26498 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:24.531424 26495 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:24.532465 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.533082 26495 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:24.398905 27890 consensus_queue.cc:1046] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20240502 08:06:24.525957 26467 leader_election.cc:398] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4.
I20240502 08:06:24.534520 26467 leader_election.cc:401] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240502 08:06:24.535010 27890 consensus_queue.cc:1046] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:24.537909 26498 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
W20240502 08:06:24.501433 27915 log.cc:924] Time spent T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.113s user 0.000s sys 0.005s
I20240502 08:06:24.538408 26501 raft_consensus.cc:2454] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:24.545418 26541 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:24.553364 26573 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:24.573434 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:24.574292 26497 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:24.552314 27845 raft_consensus.cc:3032] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:24.575132 27889 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 4.
W20240502 08:06:24.590375 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 89533.1us
I20240502 08:06:24.582538 26499 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:24.616309 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 133639us
I20240502 08:06:24.579676 27845 raft_consensus.cc:729] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
W20240502 08:06:24.628495 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 88228.7us
I20240502 08:06:24.596485 27913 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:24.632925 26615 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:24.635972 27881 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:24.636327 27832 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:24.636909 27914 raft_consensus.cc:482] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:24.708495 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 75662.8us
W20240502 08:06:24.713820 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62079.9us
W20240502 08:06:24.718437 26541 leader_election.cc:334] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.242s (SENT)
W20240502 08:06:24.707180 27930 log.cc:924] Time spent T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20240502 08:06:24.722430 26541 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:24.705945 27832 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.649443 27881 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:24.705391 27914 raft_consensus.cc:504] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.754326 27935 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 08:06:24.733696 27875 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:24.784438 26496 raft_consensus.cc:3032] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:24.784991 26496 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:24.734558 27845 consensus_queue.cc:260] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:24.791360 27876 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.502s user 0.000s sys 0.047s
I20240502 08:06:24.803413 26496 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.831624 26496 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:06:24.841095 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 264083us
I20240502 08:06:24.878449 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:24.880710 27178 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:24.882354 27963 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:06:24.882696 27956 log.cc:924] Time spent T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20240502 08:06:24.882890 27963 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.884955 27963 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:24.885483 27963 consensus_queue.cc:1046] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:24.885991 26496 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:24.887691 26495 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:24.889001 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:24.889786 26497 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:24.849643 27890 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:24.876559 27845 raft_consensus.cc:3037] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:24.876147 27881 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:24.893049 27890 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.895020 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:24.908239 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 167874us
I20240502 08:06:24.909060 26616 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:24.913130 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:24.928145 26615 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:24.910594 27832 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:24.940246 27723 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:24.942214 27914 leader_election.cc:288] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:24.960345 27892 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240502 08:06:24.979223 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 123838us
W20240502 08:06:24.942838 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 310758us
I20240502 08:06:24.942483 27145 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:24.985116 27889 raft_consensus.cc:3032] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:24.985591 27889 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:24.986450 27889 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:24.987982 27889 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:24.991923 26647 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:24.999894 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:24.983573 27723 raft_consensus.cc:686] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:25.002784 27723 consensus_queue.cc:237] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.001641 27881 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.006829 27890 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.011768 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:24.978322 27892 raft_consensus.cc:482] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:25.018224 27892 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:25.034624 26541 leader_election.cc:334] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.019s (SENT)
W20240502 08:06:25.035037 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 60437.7us
I20240502 08:06:25.056140 26570 raft_consensus.cc:1261] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240502 08:06:25.071717 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 91540.1us
I20240502 08:06:25.019567 27881 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:25.037369 27832 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240502 08:06:25.062790 27960 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.086288 26615 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.079757 27959 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240502 08:06:25.061414 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 140648us
I20240502 08:06:25.117681 27723 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 08:06:25.129048 26615 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.131974 26571 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:25.094758 27974 raft_consensus.cc:3032] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:25.076586 27970 consensus_queue.cc:1046] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20240502 08:06:25.114727 27986 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:25.082145 27960 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.145681 27974 raft_consensus.cc:729] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:25.152714 27888 raft_consensus.cc:2781] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election won for term 3
W20240502 08:06:25.196115 26541 leader_election.cc:334] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.024s (SENT)
I20240502 08:06:25.200780 26541 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:25.178706 27891 consensus_queue.cc:1046] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20240502 08:06:25.209127 27914 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240502 08:06:25.197729 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 49535.6us
W20240502 08:06:25.274370 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 51717.1us
I20240502 08:06:25.270130 27987 consensus_queue.cc:1046] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20240502 08:06:25.222913 27888 raft_consensus.cc:686] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:25.325153 27881 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:25.351828 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 124064us
I20240502 08:06:25.369158 27747 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:06:25.383114 27955 log.cc:924] Time spent T 5e0cb939b8c5477abcbd7707edbdc28b P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
W20240502 08:06:25.383708 27985 log.cc:924] Time spent T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
W20240502 08:06:25.423712 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 43877.8us
I20240502 08:06:25.423532 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:25.432065 26650 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
W20240502 08:06:25.433408 27954 log.cc:924] Time spent T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20240502 08:06:25.434505 27960 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:25.438485 27981 log.cc:924] Time spent T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20240502 08:06:25.439831 26495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1021) took 1039 ms. Trace:
I20240502 08:06:25.440286 27959 consensus_queue.cc:1046] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:25.441921 27973 raft_consensus.cc:3032] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:25.440325 26495 rpcz_store.cc:276] 0502 08:06:24.400086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.947799 (+547713us) service_pool.cc:225] Handling call
0502 08:06:24.948139 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:24.959308 (+ 11169us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:24.959354 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:24.969555 (+ 10201us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.058322 (+ 88767us) raft_consensus.cc:1631] Marking committed up to 1
0502 08:06:25.058362 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.058430 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.385056 (+326626us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4800080058. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9708fc4431 00007f9708fbb527 00007f9708fbb3ba 00007f97090af57f 00007f97090af247 00007f97090c374c 00007f97090c36f8 00007f97090c3678 00007f97090c3633 00007f97090c28dc 00007f97080459b6
0502 08:06:25.439396 (+ 54340us) raft_consensus.cc:1671] finished
0502 08:06:25.439429 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.439751 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":141901,"raft.run_cpu_time_us":1942,"raft.run_wall_time_us":294810,"spinlock_wait_cycles":9599360,"thread_start_us":85765,"threads_started":1,"wal-append.queue_time_us":109485}
I20240502 08:06:25.442334 27973 raft_consensus.cc:729] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:25.443197 27973 consensus_queue.cc:260] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.281692 27974 consensus_queue.cc:260] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.446693 27620 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.447278 27620 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.447750 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:25.448606 26495 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 2.
I20240502 08:06:25.449954 27144 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:25.451323 27963 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.451653 27881 raft_consensus.cc:2781] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:06:25.452088 27881 raft_consensus.cc:482] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:25.452422 27881 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:25.449714 27620 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:25.456974 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 104445us
W20240502 08:06:25.470744 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 390998us
W20240502 08:06:25.472244 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 193506us
I20240502 08:06:25.472920 26495 raft_consensus.cc:1261] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:25.473420 27832 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.451869 27963 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.486402 27987 consensus_queue.cc:1225] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer 97dc9560dbbe4d1aa44b2b2edda4080f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20240502 08:06:25.493995 27845 consensus_meta.cc:287] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.586s user 0.011s sys 0.000s
I20240502 08:06:25.486229 27914 raft_consensus.cc:482] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.600325 26467 leader_election.cc:398] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:25.600708 26467 leader_election.cc:401] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240502 08:06:25.601109 26500 raft_consensus.cc:1261] T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:25.393824 27888 consensus_queue.cc:237] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.607384 27992 raft_consensus.cc:3032] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:25.503815 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:25.601296 27954 log.cc:924] Time spent T e13bc17ba6de4c92a8744b92b7cd3c97 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240502 08:06:25.500823 27892 raft_consensus.cc:504] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.605324 27986 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:25.614264 27992 raft_consensus.cc:729] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:25.528976 27751 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
I20240502 08:06:25.503619 27832 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:25.645807 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 148484us
I20240502 08:06:25.625949 27178 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:25.646368 27845 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:25.646917 27983 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.648864 27969 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.604899 27914 raft_consensus.cc:504] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.631662 27962 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.665943 27969 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.666257 27962 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.668370 27969 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.668987 27948 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240502 08:06:25.670363 28002 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.670851 28002 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.671387 27914 leader_election.cc:288] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.673154 28002 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:25.644790 27986 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:25.678622 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:25.679497 26575 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 5.
I20240502 08:06:25.683110 27992 consensus_queue.cc:260] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.684687 27992 raft_consensus.cc:3037] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:25.661197 26649 raft_consensus.cc:1226] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:25.734736 27914 raft_consensus.cc:482] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.730798 27876 raft_consensus.cc:482] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.739406 27914 raft_consensus.cc:504] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.739876 27876 raft_consensus.cc:504] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.741607 27914 leader_election.cc:288] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:25.747296 27982 log.cc:924] Time spent T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240502 08:06:25.750061 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1020) took 1362 ms. Trace:
I20240502 08:06:25.750447 26499 rpcz_store.cc:276] 0502 08:06:24.387715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.941429 (+553714us) service_pool.cc:225] Handling call
0502 08:06:24.944739 (+ 3310us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:24.969650 (+ 24911us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:24.969687 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:24.972574 (+ 2887us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.064052 (+ 91478us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.079827 (+ 15775us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.372240 (+292413us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b18003098e0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f970116b2ea 00007f970a85afe5 00007f97011ef3d4 00007f97011eecc5 00007f9704114620 00007f97041267c6 00007f9704137327 00007f97041372a8 00007f9704137228
0502 08:06:25.492130 (+119890us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b18003098e0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011f21b8 00007f97011f5691 00007f97011f5649 00007f97011f55d9 00007f97011f55a1 00007f97011f489d 00007f970a466da4 00007f970a466bd9 00007f97011d4936
0502 08:06:25.607508 (+115378us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b18003098e0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011bfb06 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9708fc4431 00007f9708fbb527 00007f9708fbb3ba 00007f97090af57f
0502 08:06:25.624066 (+ 16558us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b4800080058. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9708fc4431 00007f9708fbb527 00007f9708fbb3ba 00007f97090af57f 00007f97090af247 00007f97090c374c 00007f97090c36f8 00007f97090c3678 00007f97090c3633 00007f97090c28dc 00007f97080459b6
0502 08:06:25.651252 (+ 27186us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.651366 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.749810 (+ 98444us) raft_consensus.cc:1671] finished
0502 08:06:25.749832 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.749984 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119812,"prepare.queue_time_us":17215,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":7700,"raft.queue_time_us":505598,"raft.run_cpu_time_us":10654,"raft.run_wall_time_us":538374,"spinlock_wait_cycles":139202560,"thread_start_us":369838,"threads_started":2,"wal-append.queue_time_us":174240}
I20240502 08:06:25.752861 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:25.754662 28000 log.cc:924] Time spent T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20240502 08:06:25.662135 27983 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.759682 26492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1022) took 1274 ms. Trace:
I20240502 08:06:25.759855 27963 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.729697 27845 raft_consensus.cc:482] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.760578 27845 raft_consensus.cc:504] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.760094 26492 rpcz_store.cc:276] 0502 08:06:24.485399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.981490 (+496091us) service_pool.cc:225] Handling call
0502 08:06:25.034392 (+ 52902us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.182474 (+148082us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.190565 (+ 8091us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.206773 (+ 16208us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.364910 (+158137us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.450398 (+ 85488us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.450659 (+ 261us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.450743 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.759448 (+308705us) raft_consensus.cc:1671] finished
0502 08:06:25.759474 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.759626 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62615,"prepare.queue_time_us":75095,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":380859,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"thread_start_us":177920,"threads_started":2,"wal-append.queue_time_us":278857}
I20240502 08:06:25.762794 27845 leader_election.cc:288] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:25.763157 26541 leader_election.cc:334] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.332s (SENT)
I20240502 08:06:25.687783 27986 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.765794 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:25.771440 26499 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:25.776077 27832 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:25.766624 26492 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 5: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:25.777583 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:25.778457 27177 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:25.780020 27963 consensus_queue.cc:1046] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:25.780776 27963 consensus_queue.cc:1225] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Peer 97dc9560dbbe4d1aa44b2b2edda4080f log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240502 08:06:25.781257 27992 raft_consensus.cc:3032] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:25.781754 27992 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:25.782863 27992 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.784435 27992 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:25.787053 27881 raft_consensus.cc:504] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:25.741858 27876 leader_election.cc:288] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.789505 27935 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:25.801061 26467 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.802215 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:25.805848 27993 log.cc:924] Time spent T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.453s user 0.003s sys 0.000s
I20240502 08:06:25.808624 27963 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:25.806697 27881 leader_election.cc:288] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.807644 27876 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.809907 27876 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.810747 26497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1028) took 1001 ms. Trace:
I20240502 08:06:25.811128 26497 rpcz_store.cc:276] 0502 08:06:24.809339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:25.207545 (+398206us) service_pool.cc:225] Handling call
0502 08:06:25.219752 (+ 12207us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.458616 (+238864us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.458663 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.482519 (+ 23856us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.809569 (+327050us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.810121 (+ 552us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.810479 (+ 358us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.810542 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.810581 (+ 39us) raft_consensus.cc:1671] finished
0502 08:06:25.810592 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.810700 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":332175,"prepare.queue_time_us":127,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":280002,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":2215,"spinlock_wait_cycles":1993600,"thread_start_us":308166,"threads_started":1,"wal-append.queue_time_us":321526}
I20240502 08:06:25.812754 27876 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.808944 26499 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:25.817390 26467 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:25.818689 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 174445us
I20240502 08:06:25.827451 27973 raft_consensus.cc:3037] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:25.831199 27750 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:25.835119 27968 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:25.835645 27968 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.837716 27968 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:25.840210 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:25.845705 26567 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:25.823935 26467 leader_election.cc:302] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.847888 26467 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.848891 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.841984 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:25.850376 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:25.850993 27963 raft_consensus.cc:2726] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:25.863623 26576 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:25.851028 27892 leader_election.cc:288] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:25.851670 27959 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:25.852375 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:25.865027 27963 raft_consensus.cc:2726] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:25.863624 26492 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:25.866322 26467 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.852396 26496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1027) took 1322 ms. Trace:
I20240502 08:06:25.906919 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.866688 27983 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.872000 26492 raft_consensus.cc:2454] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:25.860236 26499 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
W20240502 08:06:25.874902 27991 log.cc:924] Time spent T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.196s user 0.003s sys 0.000s
I20240502 08:06:25.876926 26498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1026) took 1347 ms. Trace:
I20240502 08:06:25.911727 26498 rpcz_store.cc:276] 0502 08:06:24.529214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:25.128704 (+599490us) service_pool.cc:225] Handling call
0502 08:06:25.150714 (+ 22010us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.151144 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.151202 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.157340 (+ 6138us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.386611 (+229271us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:25.452181 (+ 65570us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.452369 (+ 188us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.452433 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.830755 (+378322us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b1800309dc0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011f21b8 00007f97011f5691 00007f97011f5649 00007f97011f55d9 00007f97011f55a1 00007f97011f489d 00007f970a466da4 00007f970a466bd9 00007f97011d4936
0502 08:06:25.876593 (+ 45838us) raft_consensus.cc:1671] finished
0502 08:06:25.876720 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.876862 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":215224,"prepare.queue_time_us":67042,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":490,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":61,"spinlock_wait_cycles":14954240,"thread_start_us":195147,"threads_started":1,"wal-append.queue_time_us":317972}
I20240502 08:06:25.874202 27963 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:25.906172 26496 rpcz_store.cc:276] 0502 08:06:24.529509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:25.205940 (+676431us) service_pool.cc:225] Handling call
0502 08:06:25.206260 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.452971 (+246711us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.452998 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.453423 (+ 425us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.851418 (+397995us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.851942 (+ 524us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.852133 (+ 191us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.852194 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.852233 (+ 39us) raft_consensus.cc:1671] finished
0502 08:06:25.852244 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.852348 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":362902,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":279894,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":11783,"thread_start_us":397591,"threads_started":1,"wal-append.queue_time_us":367020}
I20240502 08:06:25.894584 27962 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.907641 26500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1008) took 1523 ms. Trace:
I20240502 08:06:25.907800 27959 raft_consensus.cc:2726] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:25.840451 28004 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:25.917137 26467 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.918393 26615 leader_election.cc:302] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:25.924640 26500 rpcz_store.cc:276] 0502 08:06:24.384102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.614980 (+230878us) service_pool.cc:225] Handling call
0502 08:06:24.617417 (+ 2437us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:25.601531 (+984114us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.907280 (+305749us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b5400386654. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:25.907525 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2261484800}
I20240502 08:06:25.935439 27144 leader_election.cc:302] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:25.923929 27809 raft_consensus.cc:482] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.926571 27983 raft_consensus.cc:2726] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:25.936542 27809 raft_consensus.cc:504] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.914206 27963 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.938313 27881 raft_consensus.cc:2781] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:25.938592 27809 leader_election.cc:288] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.938843 27881 raft_consensus.cc:686] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:25.939551 27963 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:25.944500 26467 leader_election.cc:302] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:25.951961 27932 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:25.952570 27932 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.956243 26494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1023) took 1450 ms. Trace:
I20240502 08:06:25.957777 27897 raft_consensus.cc:3032] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:25.958195 27897 raft_consensus.cc:729] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:25.956687 26494 rpcz_store.cc:276] 0502 08:06:24.506120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.984176 (+478056us) service_pool.cc:225] Handling call
0502 08:06:25.018477 (+ 34301us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.091253 (+ 72776us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.091314 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.091828 (+ 514us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.183003 (+ 91175us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.227279 (+ 44276us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.480697 (+253418us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.485167 (+ 4470us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.678874 (+193707us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b1800113aa0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f970a093661 00007f970a0a03e1 00007f97011f21b8 00007f97011f5691 00007f97011f5649 00007f97011f55d9 00007f97011f55a1 00007f97011f489d 00007f970a466da4 00007f970a466bd9 00007f97011d4936
0502 08:06:25.835097 (+156223us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b44000232cc. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f9701106685 00007f97011070c2 00007f9703daf441 00007f9703daad4a 00007f9703d9eb1e 00007f9703d9de66 00007f9703d9da88 00007f9703da7481 00007f9703da7439 00007f9703da73c9
0502 08:06:25.956016 (+120919us) raft_consensus.cc:1671] finished
0502 08:06:25.956051 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.956183 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":331029,"prepare.queue_time_us":23871,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":792650,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":6326,"spinlock_wait_cycles":114450560,"thread_start_us":69017,"threads_started":1,"wal-append.queue_time_us":175430}
I20240502 08:06:25.934161 27891 raft_consensus.cc:2781] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:25.959509 27891 raft_consensus.cc:686] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:25.960369 27891 consensus_queue.cc:237] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:25.962886 27932 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:25.963451 28001 log.cc:924] Time spent T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.607s user 0.005s sys 0.013s
I20240502 08:06:25.965444 26493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1025) took 1436 ms. Trace:
I20240502 08:06:25.965804 26493 rpcz_store.cc:276] 0502 08:06:24.528949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:25.032940 (+503991us) service_pool.cc:225] Handling call
0502 08:06:25.033323 (+ 383us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:25.176229 (+142906us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:25.182124 (+ 5895us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:25.217681 (+ 35557us) log.cc:842] Serialized 26 byte log entry
0502 08:06:25.364892 (+147211us) raft_consensus.cc:1631] Marking committed up to 3
0502 08:06:25.429816 (+ 64924us) log.cc:842] Serialized 14 byte log entry
0502 08:06:25.500049 (+ 70233us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:25.504422 (+ 4373us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:25.962820 (+458398us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b44000232cc. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f9701106685 00007f97011070c2 00007f9703daf441 00007f9703daad4a 00007f9703d9eb1e 00007f9703d9de66 00007f9703d9da88 00007f9703da7481 00007f9703da7439 00007f9703da73c9
0502 08:06:25.965108 (+ 2288us) raft_consensus.cc:1671] finished
0502 08:06:25.965223 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:25.965384 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121323,"prepare.queue_time_us":82395,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":583615,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":159,"spinlock_wait_cycles":3657472,"thread_start_us":145409,"threads_started":2,"wal-append.queue_time_us":123210}
I20240502 08:06:25.968679 28012 consensus_queue.cc:1035] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:25.974644 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:25.975499 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:25.975571 26495 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:25.977123 26575 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:25.980497 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:25.939754 27881 consensus_queue.cc:237] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:25.988611 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:25.943643 27962 raft_consensus.cc:2726] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.010723 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.018229 27990 raft_consensus.cc:3032] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:26.018886 27990 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:26.020006 27990 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.021631 27990 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:26.024245 26494 raft_consensus.cc:1226] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:26.025642 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aba3e3cbd94435eabac55bb87284652" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:26.032431 26399 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52756 (request call id 23) took 3253 ms. Trace:
I20240502 08:06:26.036659 27974 raft_consensus.cc:3037] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:26.043136 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 98738.4us
I20240502 08:06:26.043320 27973 raft_consensus.cc:2726] T d73e56c17a6f4770a16b443ab75ea198 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.043910 26467 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.044126 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:26.044891 27751 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:26.045001 26467 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.045327 27751 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:26.045610 28005 raft_consensus.cc:2726] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.045768 27751 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.046182 28004 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.047036 26467 leader_election.cc:302] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.054792 26616 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:26.055018 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.056445 27935 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 08:06:26.056847 27935 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:26.057169 27935 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.058462 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:26.059222 26644 raft_consensus.cc:595] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.059274 27897 consensus_queue.cc:260] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.060539 27897 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:26.062523 27962 raft_consensus.cc:2726] T 77b2256f7eb5420792d8042fb8e30d0b P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.063685 26616 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:26.065524 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.065878 27905 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240502 08:06:26.066474 27905 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:26.066812 26647 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:26.066359 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.069099 26467 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.070501 26649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55496 (request call id 563) took 1078 ms. Trace:
I20240502 08:06:26.072072 27948 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:26.066877 27905 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.075371 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.032985 26399 rpcz_store.cc:276] 0502 08:06:22.779390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:22.779607 (+ 217us) service_pool.cc:225] Handling call
0502 08:06:26.032355 (+3252748us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:24.075868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:24.125859 (+ 49991us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:24.125900 (+ 41us) write_op.cc:427] Acquired schema lock
0502 08:06:24.125919 (+ 19us) tablet.cc:660] Decoding operations
0502 08:06:24.126274 (+ 355us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:24.126334 (+ 60us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:24.126362 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:06:24.126541 (+ 179us) tablet.cc:699] Row locks acquired
0502 08:06:24.126556 (+ 15us) write_op.cc:260] PREPARE: finished
0502 08:06:24.126690 (+ 134us) write_op.cc:270] Start()
0502 08:06:24.126786 (+ 96us) write_op.cc:276] Timestamp: P: 1714637184126648 usec, L: 0
0502 08:06:24.126801 (+ 15us) op_driver.cc:331] REPLICATION: starting
0502 08:06:24.194757 (+ 67956us) log.cc:842] Serialized 594 byte log entry
0502 08:06:24.878029 (+683272us) op_driver.cc:444] REPLICATION: finished
0502 08:06:25.144111 (+266082us) write_op.cc:301] APPLY: starting
0502 08:06:25.149119 (+ 5008us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:25.175428 (+ 26309us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:25.190373 (+ 14945us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:25.199933 (+ 9560us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:25.222247 (+ 22314us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:06:25.226750 (+ 4503us) write_op.cc:312] APPLY: finished
0502 08:06:25.266849 (+ 40099us) log.cc:842] Serialized 22 byte log entry
0502 08:06:25.328623 (+ 61774us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:25.340459 (+ 11836us) write_op.cc:446] Released schema lock
0502 08:06:25.342643 (+ 2184us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:25.357316 (+ 14673us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:25.364188 (+ 6872us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":152811,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":225722,"apply.run_cpu_time_us":3367,"apply.run_wall_time_us":242283,"num_ops":1,"prepare.queue_time_us":79679,"prepare.run_cpu_time_us":3080,"prepare.run_wall_time_us":162320,"replication_time_us":728278,"thread_start_us":137862,"threads_started":2}]]}
I20240502 08:06:26.086771 26467 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.081342 27962 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.106289 26497 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:26.130836 26467 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.133148 26542 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:26.148639 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.149647 26647 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5216c4a3a33f47ac88fa35704487f77e in current term 5: Already voted for candidate 755f35bc66a841f382d1c4df6c6f6b64 in this term.
I20240502 08:06:26.148823 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a26db3828484577a5da525732f0b102" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:26.154907 26645 raft_consensus.cc:3037] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:26.126981 27983 raft_consensus.cc:2726] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.158156 26395 catalog_manager.cc:5484] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.163491 26395 catalog_manager.cc:5484] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.172613 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.168736 26395 catalog_manager.cc:5484] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.150560 27992 raft_consensus.cc:2726] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.175561 26395 catalog_manager.cc:5484] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.079442 26649 rpcz_store.cc:276] 0502 08:06:24.991705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.991869 (+ 164us) service_pool.cc:225] Handling call
0502 08:06:24.992184 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:25.739253 (+747069us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:26.070247 (+330994us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400140d54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:26.070425 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1518891264}
I20240502 08:06:26.183800 26395 catalog_manager.cc:5484] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.188380 26395 catalog_manager.cc:5484] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.192871 26467 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.171252 27919 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240502 08:06:26.199573 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 59578.5us
I20240502 08:06:26.190822 27892 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240502 08:06:26.202401 26541 leader_election.cc:334] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.094s (SENT)
I20240502 08:06:26.192601 26395 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index 2 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.204589 26395 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.198623 27919 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:26.206753 26395 catalog_manager.cc:5484] T 3749d35fdedd4d979e761e8b7f57ad39 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.211592 26395 catalog_manager.cc:5484] T 00f1f297bff34c27b4789483f0b3668f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.214061 26395 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.200912 27992 raft_consensus.cc:2726] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.216022 26395 catalog_manager.cc:5484] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.233444 26395 catalog_manager.cc:5484] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
W20240502 08:06:26.238852 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 116974us
I20240502 08:06:26.235368 26395 catalog_manager.cc:5484] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.240448 26395 catalog_manager.cc:5484] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:26.202723 27892 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 97dc9560dbbe4d1aa44b2b2edda4080f)
I20240502 08:06:26.245810 26645 raft_consensus.cc:2454] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 3.
I20240502 08:06:26.242933 26395 catalog_manager.cc:5484] T d73e56c17a6f4770a16b443ab75ea198 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.209769 27919 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.185782 27145 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:26.248328 27892 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.216145 27832 consensus_queue.cc:1046] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20240502 08:06:26.285285 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 82923.9us
I20240502 08:06:26.249982 26395 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.344094 26467 leader_election.cc:398] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3.
I20240502 08:06:26.349570 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:26.350286 26573 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:26.345279 26467 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.351719 26467 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:26.317489 28023 log.cc:924] Time spent T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240502 08:06:26.344038 26395 catalog_manager.cc:5484] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.280990 27881 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.359422 26395 catalog_manager.cc:5484] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.361416 26395 catalog_manager.cc:5484] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
W20240502 08:06:26.316596 26541 leader_election.cc:334] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 3.249s (SENT)
I20240502 08:06:26.373014 26541 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:26.367215 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 138477us
I20240502 08:06:26.372938 26395 catalog_manager.cc:5484] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:26.356781 27992 raft_consensus.cc:2726] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.356904 27973 raft_consensus.cc:2726] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:26.347772 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 173591us
I20240502 08:06:26.387058 27809 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.390280 27881 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240502 08:06:26.390995 28006 log.cc:924] Time spent T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20240502 08:06:26.356694 27935 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:26.399971 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 83414.6us
I20240502 08:06:26.381134 26395 catalog_manager.cc:5484] T 39e50c6374394235996663b59b4be48a P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
W20240502 08:06:26.443856 28023 log.cc:924] Time spent T 82da19e5ed7d415588714d971127c197 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20240502 08:06:26.516709 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 205554us
I20240502 08:06:26.603279 26395 catalog_manager.cc:5484] T 82da19e5ed7d415588714d971127c197 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:26.605129 27935 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
W20240502 08:06:26.606861 28017 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e: Append to log took a long time: real 0.347s user 0.003s sys 0.012s
I20240502 08:06:26.605983 26395 catalog_manager.cc:5484] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
W20240502 08:06:26.608215 27905 consensus_meta.cc:287] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.522s user 0.013s sys 0.005s
I20240502 08:06:26.638422 27905 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.608799 26395 catalog_manager.cc:5484] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:26.640753 27905 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:26.611778 27809 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:26.615057 28014 raft_consensus.cc:3032] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:26.641450 28014 raft_consensus.cc:729] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:26.641523 27809 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.642089 26649 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:26.642411 28014 consensus_queue.cc:260] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.643481 28014 raft_consensus.cc:3037] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:26.643625 27809 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.645118 26650 raft_consensus.cc:1226] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:26.644944 27881 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:06:26.617518 26541 leader_election.cc:334] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.691s (SENT)
W20240502 08:06:26.614231 28028 log.cc:924] Time spent T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.232s user 0.009s sys 0.005s
W20240502 08:06:26.620417 28024 log.cc:924] Time spent T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.302s user 0.004s sys 0.011s
W20240502 08:06:26.622334 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 223246us
I20240502 08:06:26.653352 26495 raft_consensus.cc:1261] T 9a26db3828484577a5da525732f0b102 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:26.627700 27892 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.629657 26501 raft_consensus.cc:1261] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:26.632153 27875 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
W20240502 08:06:26.635454 27990 consensus_meta.cc:287] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.558s user 0.011s sys 0.006s
W20240502 08:06:26.637805 27751 consensus_meta.cc:287] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: Time spent flushing consensus metadata: real 0.592s user 0.011s sys 0.004s
W20240502 08:06:26.651378 26541 leader_election.cc:334] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.791s (SENT)
I20240502 08:06:26.666544 26541 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:26.666900 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:26.667500 26495 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.670424 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.671267 26645 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:26.671820 26644 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 5. Current term is 6. Ops: []
I20240502 08:06:26.675861 26501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60752 (request call id 1035) took 1184 ms. Trace:
I20240502 08:06:26.675978 26467 leader_election.cc:302] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.676333 26501 rpcz_store.cc:276] 0502 08:06:25.491837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:25.631588 (+139751us) service_pool.cc:225] Handling call
0502 08:06:25.631892 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0502 08:06:26.655056 (+1023164us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:26.675634 (+ 20578us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54000b1b54. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f970411d1a1 00007f970411bbee 00007f970905e099 00007f9702740694 00007f9702740622 00007f9702740551 00007f97027404cc 00007f970273f782 00007f9702543bcc 00007f9702543006
0502 08:06:26.675806 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1768729984}
I20240502 08:06:26.658535 27983 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:26.662948 27974 raft_consensus.cc:482] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:26.665901 27978 raft_consensus.cc:482] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:26.697173 26467 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.677886 27990 raft_consensus.cc:482] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:26.661823 27919 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:26.665879 27875 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.678438 27881 consensus_queue.cc:1035] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240502 08:06:26.704747 27875 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:26.708679 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 43771.2us
I20240502 08:06:26.669752 27751 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.698027 27962 raft_consensus.cc:2726] T d0b9fcbac5f24ae8b0c8f071ae34229f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.717839 26467 leader_election.cc:302] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.719151 26542 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:26.719597 26499 raft_consensus.cc:1261] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:26.723726 26495 raft_consensus.cc:2396] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755f35bc66a841f382d1c4df6c6f6b64 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240502 08:06:26.708997 27973 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.685334 27983 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.710845 27809 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.734349 28014 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240502 08:06:26.742524 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 91166.9us
I20240502 08:06:26.748391 26574 raft_consensus.cc:1261] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:26.687346 27974 raft_consensus.cc:504] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:26.701174 27978 raft_consensus.cc:504] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.752261 27973 raft_consensus.cc:2726] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.753479 26467 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.708127 27990 raft_consensus.cc:504] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.766673 26499 raft_consensus.cc:1261] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:26.733304 27832 consensus_queue.cc:1046] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20240502 08:06:26.743868 28014 raft_consensus.cc:482] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:26.766217 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.754385 27870 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:26.771167 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:26.772044 26498 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.784200 28041 raft_consensus.cc:3032] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:26.784617 28041 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:26.785276 28041 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.767681 27962 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.786192 28041 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.779906 27870 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.789065 27919 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:26.789260 27870 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.790380 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:26.797904 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.773978 28014 raft_consensus.cc:3037] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:26.773849 27892 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:26.807958 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:26.806573 26498 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
I20240502 08:06:26.808794 26644 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.810829 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:26.811504 26493 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.815491 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.818998 27973 raft_consensus.cc:2726] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.801826 27178 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.823499 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.826887 27978 leader_election.cc:288] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.828280 27934 raft_consensus.cc:979] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:26.828811 27879 raft_consensus.cc:1067] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:26.828779 27934 consensus_queue.cc:1035] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:06:26.824335 26541 leader_election.cc:334] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.062s (SENT)
W20240502 08:06:26.832365 26541 leader_election.cc:334] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.084s (SENT)
I20240502 08:06:26.840723 27962 raft_consensus.cc:482] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:26.841293 27962 raft_consensus.cc:504] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.843361 27962 leader_election.cc:288] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.845973 27178 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 6.
W20240502 08:06:26.850327 28036 log.cc:924] Time spent T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20240502 08:06:26.850796 27974 leader_election.cc:288] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:26.851949 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:26.822639 27983 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.854357 27978 raft_consensus.cc:2726] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.857909 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:26.856549 26573 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.860435 27620 consensus_queue.cc:1035] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:26.861042 27620 consensus_queue.cc:1225] T e13bc17ba6de4c92a8744b92b7cd3c97 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer 8f9fe1f3763547388c4301b13a127aaa log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240502 08:06:26.866425 27990 leader_election.cc:288] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:26.835117 27751 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:26.867117 27968 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:26.872109 26615 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:26.873265 27887 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:26.873890 27887 raft_consensus.cc:686] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:26.874954 27887 consensus_queue.cc:237] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.863602 26644 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 6.
I20240502 08:06:26.855675 26467 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.883163 26493 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
I20240502 08:06:26.862150 27620 consensus_queue.cc:1035] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20240502 08:06:26.877190 26573 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:26.877890 28014 raft_consensus.cc:504] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.890523 27990 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:26.893723 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:26.912578 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 57444.8us
I20240502 08:06:26.919220 26615 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:26.909819 26493 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:26.928159 26467 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.942955 26467 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.944716 26467 leader_election.cc:302] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.944989 26494 pending_rounds.cc:76] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Aborting all ops after (but not including) 1
I20240502 08:06:26.945327 26494 pending_rounds.cc:97] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240502 08:06:26.945454 27978 raft_consensus.cc:2726] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:26.945600 26494 raft_consensus.cc:2866] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:26.946137 27978 raft_consensus.cc:2726] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.946765 27978 raft_consensus.cc:2726] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:06:26.948051 26467 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.949611 27824 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:26.950168 27824 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:26.951040 27824 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:26.952101 27983 raft_consensus.cc:2726] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:26.953213 26467 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:26.958335 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.960574 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.966603 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.972525 26646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.973054 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.975642 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.962756 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:26.982550 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:26.983990 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:26.984083 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:26.984562 26568 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:26.988152 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:26.989424 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:26.999012 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:26.999913 26567 raft_consensus.cc:2379] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 3: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:26.999783 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:27.001111 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.004637 26569 raft_consensus.cc:1226] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:27.005496 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b2f57a1a34420e98d2d4d7f2f9162e" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.009032 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.010164 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.014067 26573 raft_consensus.cc:1261] T 6aba3e3cbd94435eabac55bb87284652 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:26.978801 28048 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:26.981498 27978 raft_consensus.cc:2726] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.024075 26572 raft_consensus.cc:1226] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 3. Current term is 5. Ops: [3.2-3.2]
I20240502 08:06:27.039180 26645 raft_consensus.cc:2454] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 3.
I20240502 08:06:27.040903 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.013859 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:27.042130 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:27.044240 26571 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:27.048086 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.022318 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.051532 28014 leader_election.cc:288] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.057960 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.059917 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
W20240502 08:06:27.022794 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 176796us
W20240502 08:06:27.089874 28021 log.cc:924] Time spent T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240502 08:06:27.115217 26616 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.112643 26574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
W20240502 08:06:27.121635 28046 log.cc:924] Time spent T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20240502 08:06:27.117712 26574 raft_consensus.cc:482] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:27.124800 26574 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:27.130162 27935 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:27.136243 26493 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:27.032217 28048 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:27.142969 26570 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
I20240502 08:06:27.111943 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.085280 26568 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
I20240502 08:06:27.151564 26571 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:27.154765 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.158722 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.158711 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.159950 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.158753 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13bc17ba6de4c92a8744b92b7cd3c97" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.160456 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.161271 26571 raft_consensus.cc:2354] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 5. Current term is 6.
I20240502 08:06:27.161211 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.156612 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.162082 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.162784 26569 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 6: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:27.162909 26500 raft_consensus.cc:2379] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:27.162340 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
W20240502 08:06:27.160014 26541 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.978s (SENT) [suppressed 37 similar messages]
I20240502 08:06:27.162287 26496 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:27.162992 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.167084 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.167472 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.167798 26497 raft_consensus.cc:2354] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5.
I20240502 08:06:27.168193 26498 raft_consensus.cc:2354] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5.
I20240502 08:06:27.168546 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.169162 26645 raft_consensus.cc:2379] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa in current term 6: Already voted for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in this term.
I20240502 08:06:27.169574 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.169803 26498 raft_consensus.cc:1226] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:27.171811 27935 raft_consensus.cc:686] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:27.172194 27542 consensus_queue.cc:1035] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:27.172727 27935 consensus_queue.cc:237] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.175984 28048 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.179580 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.180128 26570 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:27.182871 27620 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:27.184171 27620 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.186653 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.186194 26574 raft_consensus.cc:504] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.187323 26498 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:27.206602 26574 leader_election.cc:288] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.206637 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.207170 27144 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:27.186978 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.221004 26649 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 5.
I20240502 08:06:27.207875 26493 raft_consensus.cc:1261] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:27.209561 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.223515 26498 raft_consensus.cc:3037] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:27.210566 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:27.212967 26495 raft_consensus.cc:1226] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:27.214669 26496 raft_consensus.cc:1203] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20240502 08:06:27.215740 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:27.218937 26499 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:27.224941 26642 raft_consensus.cc:3032] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:27.225071 26496 raft_consensus.cc:1226] T 6aba3e3cbd94435eabac55bb87284652 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240502 08:06:27.225773 26642 raft_consensus.cc:729] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:27.228886 26642 consensus_queue.cc:260] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.229601 27978 raft_consensus.cc:482] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:27.233381 27978 raft_consensus.cc:504] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.235353 27978 leader_election.cc:288] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.236205 26642 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:27.226622 28045 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240502 08:06:27.237900 28045 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:27.238799 28045 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:27.253261 26642 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:27.254546 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.270542 26496 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:27.232786 27879 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:27.272634 27879 raft_consensus.cc:686] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:27.265622 26498 raft_consensus.cc:2454] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:27.266831 26497 raft_consensus.cc:2454] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:27.273496 27879 consensus_queue.cc:237] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.281131 26542 leader_election.cc:302] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:27.282985 26497 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:27.293571 28041 raft_consensus.cc:2781] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:27.294117 28041 raft_consensus.cc:686] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:27.300112 26497 raft_consensus.cc:1261] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:27.295295 28041 consensus_queue.cc:237] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.307962 26497 raft_consensus.cc:1203] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 08:06:27.308658 26497 raft_consensus.cc:1226] T 147d10cad3be4380a85513c7cbaef19e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240502 08:06:27.292490 28045 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:27.310137 28050 log.cc:924] Time spent T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240502 08:06:27.319195 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.319634 26399 catalog_manager.cc:5484] T 9a26db3828484577a5da525732f0b102 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 1 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4). New cstate: current_term: 3 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:27.321053 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.321525 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:27.322163 26568 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 6
I20240502 08:06:27.324635 26497 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 6
I20240502 08:06:27.329623 27978 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:27.297039 27620 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:27.330163 27978 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.332078 27978 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.333652 26615 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
W20240502 08:06:27.333761 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 173766us
I20240502 08:06:27.333096 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:27.335124 26649 raft_consensus.cc:3037] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:27.335160 26541 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:27.335865 26568 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 6.
I20240502 08:06:27.326215 28045 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 6 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:27.342005 28035 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:06:27.343613 28035 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:27.343987 28035 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:27.352100 26497 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 6.
I20240502 08:06:27.342484 27875 raft_consensus.cc:2781] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election won for term 6
W20240502 08:06:27.358412 26541 leader_election.cc:334] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.766s (SENT)
I20240502 08:06:27.371081 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.371436 26649 raft_consensus.cc:2454] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 6.
I20240502 08:06:27.371978 26573 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 3.
I20240502 08:06:27.389369 26571 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:27.392798 26569 raft_consensus.cc:1226] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:27.400750 26495 raft_consensus.cc:1261] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:27.422307 26498 raft_consensus.cc:1226] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 2. Current term is 3. Ops: []
I20240502 08:06:27.424034 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.395659 27875 raft_consensus.cc:686] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:27.384083 27144 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:27.432699 26571 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:27.433334 26496 raft_consensus.cc:2354] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 5. Current term is 6.
I20240502 08:06:27.437355 26495 raft_consensus.cc:1226] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Rejecting Update request from peer 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 5. Current term is 6. Ops: []
W20240502 08:06:27.438792 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 51542.5us
I20240502 08:06:27.439671 26498 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:06:27.441118 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.443405 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.444298 26495 raft_consensus.cc:2354] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5.
I20240502 08:06:27.445968 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.447227 26495 raft_consensus.cc:2354] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 5. Current term is 6.
I20240502 08:06:27.449378 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
W20240502 08:06:27.440798 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 82670.3us
I20240502 08:06:27.450564 26495 raft_consensus.cc:2362] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5. Re-sending same reply.
I20240502 08:06:27.453706 26496 raft_consensus.cc:2354] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 5. Current term is 6.
I20240502 08:06:27.458421 27875 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:27.450520 28045 raft_consensus.cc:2781] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:27.459851 28045 raft_consensus.cc:686] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 6 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:27.461256 28045 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:27.433909 26494 raft_consensus.cc:1261] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:27.468575 26499 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:27.475016 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.475149 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.476346 26496 raft_consensus.cc:2354] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 4. Current term is 5.
I20240502 08:06:27.480517 26495 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:27.483152 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.496040 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.496598 26495 raft_consensus.cc:2362] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3. Re-sending same reply.
I20240502 08:06:27.499413 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0cb939b8c5477abcbd7707edbdc28b" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.500196 26501 raft_consensus.cc:2362] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5. Re-sending same reply.
I20240502 08:06:27.500299 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.501793 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.499109 27978 raft_consensus.cc:482] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:27.510398 27542 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.005s
I20240502 08:06:27.515677 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.516309 26498 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:27.516937 28012 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20240502 08:06:27.517561 28012 consensus_queue.cc:1225] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer 8f9fe1f3763547388c4301b13a127aaa log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240502 08:06:27.521112 26499 pending_rounds.cc:76] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Aborting all ops after (but not including) 2
I20240502 08:06:27.521435 26499 pending_rounds.cc:97] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240502 08:06:27.521850 26499 raft_consensus.cc:2866] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:27.518828 27978 raft_consensus.cc:504] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.511176 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.523948 27978 leader_election.cc:288] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
W20240502 08:06:27.525766 26541 leader_election.cc:334] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 1.889s (SENT)
I20240502 08:06:27.536278 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.537948 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.538964 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d10cad3be4380a85513c7cbaef19e" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.538110 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.541124 26497 raft_consensus.cc:2379] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:27.542022 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3749d35fdedd4d979e761e8b7f57ad39" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.542392 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.544543 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.545243 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
W20240502 08:06:27.547093 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 170712us
W20240502 08:06:27.547627 26541 leader_election.cc:334] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): Timed out: RequestConsensusVote RPC to 127.24.194.1:40577 timed out after 2.107s (SENT)
I20240502 08:06:27.548103 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d53d3e99887443b8967129ddd219819" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.560053 26541 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.560855 28068 raft_consensus.cc:2726] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.561451 28035 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.563601 28035 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:27.560178 26494 raft_consensus.cc:3037] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:27.564524 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.595086 26498 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:27.570416 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba760137cc473c9fefe88a2cc261f0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.555613 26398 catalog_manager.cc:5484] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:27.571883 26501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.572203 26497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.573415 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e50c6374394235996663b59b4be48a" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.596976 26398 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:27.573729 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14320b2cc936478c8fe96bc8361ad980" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:27.602394 26495 raft_consensus.cc:3037] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:27.604228 26493 raft_consensus.cc:2379] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f in current term 3: Already voted for candidate 8f9fe1f3763547388c4301b13a127aaa in this term.
I20240502 08:06:27.574704 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc168bb1b4064b7e7801a4f94f5ff" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:27.578269 28014 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:27.585829 26649 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:27.586854 26395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39274 (request call id 139) took 3376 ms. Trace:
I20240502 08:06:27.613075 26498 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:27.614631 26615 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:27.616914 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.622684 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d753d3fcb9249c28f415af2ab615999" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.611006 26395 rpcz_store.cc:276] 0502 08:06:24.210363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:24.211357 (+ 994us) service_pool.cc:225] Handling call
0502 08:06:24.237888 (+ 26531us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b54000037d4. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a1d30 00007f970412517a 00007f970a076b1c 00007f970a170a8e 00007f9708484a44 00007f97084849d2 00007f9708484901 00007f970848487c 00007f9708483b32 00007f9702543bcc 00007f9702543006
0502 08:06:27.586793 (+3348905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:26.835683 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:26.837270 (+ 1587us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:26.837309 (+ 39us) write_op.cc:427] Acquired schema lock
0502 08:06:26.837327 (+ 18us) tablet.cc:660] Decoding operations
0502 08:06:26.839836 (+ 2509us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:26.839914 (+ 78us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:26.839941 (+ 27us) tablet.cc:683] Acquiring locks for 25 operations
0502 08:06:26.842136 (+ 2195us) tablet.cc:699] Row locks acquired
0502 08:06:26.842157 (+ 21us) write_op.cc:260] PREPARE: finished
0502 08:06:26.842295 (+ 138us) write_op.cc:270] Start()
0502 08:06:26.842403 (+ 108us) write_op.cc:276] Timestamp: P: 1714637186842253 usec, L: 0
0502 08:06:26.842426 (+ 23us) op_driver.cc:331] REPLICATION: starting
0502 08:06:26.843110 (+ 684us) log.cc:842] Serialized 9831 byte log entry
0502 08:06:27.177737 (+334627us) op_driver.cc:444] REPLICATION: finished
0502 08:06:27.221602 (+ 43865us) write_op.cc:301] APPLY: starting
0502 08:06:27.221679 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:27.228777 (+ 7098us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:27.228906 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:27.291514 (+ 62608us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:27.291818 (+ 304us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=25
0502 08:06:27.291862 (+ 44us) write_op.cc:312] APPLY: finished
0502 08:06:27.292899 (+ 1037us) log.cc:842] Serialized 169 byte log entry
0502 08:06:27.293078 (+ 179us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:27.293271 (+ 193us) write_op.cc:446] Released schema lock
0502 08:06:27.293332 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:27.293583 (+ 251us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:27.293613 (+ 30us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":81,"reported_tablets":43,"spinlock_wait_cycles":31302400,"child_traces":[["op",{"apply.queue_time_us":43633,"apply.run_cpu_time_us":6458,"apply.run_wall_time_us":72650,"num_ops":25,"prepare.queue_time_us":126805,"prepare.run_cpu_time_us":26393,"prepare.run_wall_time_us":77266,"raft.queue_time_us":55365,"raft.run_cpu_time_us":19393,"raft.run_wall_time_us":31497,"replication_time_us":335248,"thread_start_us":246170,"threads_started":4,"wal-append.queue_time_us":79386}]]}
I20240502 08:06:27.632409 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.633535 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.639215 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:27.640656 26573 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:27.649263 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:27.635000 28035 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election won for term 4
W20240502 08:06:27.652526 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 197460us
I20240502 08:06:27.656636 26467 leader_election.cc:398] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 5. Current term is 6.
I20240502 08:06:27.656901 26467 leader_election.cc:401] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:27.657482 26467 leader_election.cc:302] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.659373 26467 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.672261 26495 raft_consensus.cc:2454] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:27.636139 28069 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240502 08:06:27.663100 28035 raft_consensus.cc:686] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:27.678391 28035 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.685312 27905 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:27.687556 27948 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:27.637367 28041 consensus_queue.cc:1035] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240502 08:06:27.642931 28067 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20240502 08:06:27.735671 27178 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240502 08:06:27.717993 28046 log.cc:924] Time spent T 00000000000000000000000000000000 P 8d0dec82658b4326be63df825d64209e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240502 08:06:27.744009 26498 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:27.755661 28038 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:27.759469 26494 raft_consensus.cc:2454] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 6.
I20240502 08:06:27.760577 27983 raft_consensus.cc:2726] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 08:06:27.749020 27978 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601) with higher term. Message: Invalid argument: T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f9fe1f3763547388c4301b13a127aaa for earlier term 5. Current term is 6.
W20240502 08:06:27.774137 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 114836us
I20240502 08:06:27.781566 26467 leader_election.cc:302] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:27.781885 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 143340us
I20240502 08:06:27.782621 26541 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.799826 26541 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.802350 26541 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.804181 26541 leader_election.cc:302] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.783536 28070 raft_consensus.cc:2726] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.812572 28041 raft_consensus.cc:2726] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:27.834003 28081 raft_consensus.cc:2726] T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.819837 28067 consensus_queue.cc:1035] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240502 08:06:27.823724 28070 raft_consensus.cc:2726] T 147d10cad3be4380a85513c7cbaef19e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.801826 27875 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240502 08:06:27.839282 26495 raft_consensus.cc:1261] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240502 08:06:27.839689 26495 pending_rounds.cc:76] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Aborting all ops after (but not including) 1
I20240502 08:06:27.839920 26495 pending_rounds.cc:97] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240502 08:06:27.840211 26495 raft_consensus.cc:2866] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:06:27.821189 28069 raft_consensus.cc:2726] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:27.768868 28037 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240502 08:06:27.843600 26541 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.844686 28087 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.800534 28068 consensus_queue.cc:1035] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240502 08:06:27.845161 26541 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.845901 28087 raft_consensus.cc:2726] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.846333 26541 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.846971 28087 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.847378 26541 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:27.848004 28087 raft_consensus.cc:2726] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.821830 28082 raft_consensus.cc:2726] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:06:27.855307 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 73833.4us
I20240502 08:06:27.856015 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.856603 28082 raft_consensus.cc:2726] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.857640 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.858856 26467 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.860050 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.860038 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:27.860944 26568 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:27.858382 28082 raft_consensus.cc:2726] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.864430 28084 raft_consensus.cc:2726] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.867522 26572 raft_consensus.cc:1261] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240502 08:06:27.874742 26568 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 4.
I20240502 08:06:27.879871 26644 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:27.893698 28079 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240502 08:06:27.904754 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 43711.5us
I20240502 08:06:27.905499 26467 leader_election.cc:302] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.911490 26395 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:27.922703 26467 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.920856 28032 consensus_queue.cc:1035] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240502 08:06:27.927464 26573 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:27.919649 28082 raft_consensus.cc:2726] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.931641 26575 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240502 08:06:27.933415 27983 raft_consensus.cc:2726] T 12b2f57a1a34420e98d2d4d7f2f9162e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:27.936348 27177 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:27.945374 27983 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 08:06:27.943949 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.951798 26467 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.952924 26467 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.953554 28085 raft_consensus.cc:2726] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 08:06:27.953919 27978 raft_consensus.cc:2726] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.957707 27978 raft_consensus.cc:2781] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:27.958288 27978 raft_consensus.cc:686] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:27.959281 27978 consensus_queue.cc:237] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:27.972470 26467 leader_election.cc:302] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.978907 26467 leader_election.cc:302] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.980558 26467 leader_election.cc:302] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.981216 26495 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240502 08:06:27.984282 28082 raft_consensus.cc:2726] T 3749d35fdedd4d979e761e8b7f57ad39 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.979782 28085 raft_consensus.cc:2726] T e13bc17ba6de4c92a8744b92b7cd3c97 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 08:06:27.987502 26467 leader_election.cc:302] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:27.986243 27983 raft_consensus.cc:2726] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:28.000932 26396 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:28.007416 28085 raft_consensus.cc:2726] T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:28.016834 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f1f297bff34c27b4789483f0b3668f" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:28.019836 26398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39264 (request call id 130) took 1056 ms. Trace:
I20240502 08:06:28.020627 26493 raft_consensus.cc:1261] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:28.022333 27609 consensus_queue.cc:1035] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:28.023957 26497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33e1db2e4cc44834966a5a186294dff8"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:28.024538 26497 raft_consensus.cc:595] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.024863 28045 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:28.024891 26497 raft_consensus.cc:603] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 08:06:28.019917 26396 catalog_manager.cc:5484] T 1d753d3fcb9249c28f415af2ab615999 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:28.028729 26467 leader_election.cc:302] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.030517 26396 catalog_manager.cc:5484] T 5e0cb939b8c5477abcbd7707edbdc28b P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 3 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:28.035715 26541 leader_election.cc:302] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.035763 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:28.033485 26396 catalog_manager.cc:5484] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 3, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:28.020409 26398 rpcz_store.cc:276] 0502 08:06:26.963862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:26.964067 (+ 205us) service_pool.cc:225] Handling call
0502 08:06:28.019776 (+1055709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:27.607260 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:27.607735 (+ 475us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:27.607772 (+ 37us) write_op.cc:427] Acquired schema lock
0502 08:06:27.607798 (+ 26us) tablet.cc:660] Decoding operations
0502 08:06:27.608336 (+ 538us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:27.608411 (+ 75us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:27.608452 (+ 41us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:06:27.608854 (+ 402us) tablet.cc:699] Row locks acquired
0502 08:06:27.608881 (+ 27us) write_op.cc:260] PREPARE: finished
0502 08:06:27.609068 (+ 187us) write_op.cc:270] Start()
0502 08:06:27.609196 (+ 128us) write_op.cc:276] Timestamp: P: 1714637187609013 usec, L: 0
0502 08:06:27.609220 (+ 24us) op_driver.cc:331] REPLICATION: starting
0502 08:06:27.648595 (+ 39375us) log.cc:842] Serialized 922 byte log entry
0502 08:06:27.761222 (+112627us) op_driver.cc:444] REPLICATION: finished
0502 08:06:27.761631 (+ 409us) write_op.cc:301] APPLY: starting
0502 08:06:27.761713 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:27.761820 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:27.761907 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:27.762226 (+ 319us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:27.762375 (+ 149us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:06:27.762416 (+ 41us) write_op.cc:312] APPLY: finished
0502 08:06:27.762599 (+ 183us) log.cc:842] Serialized 28 byte log entry
0502 08:06:27.762749 (+ 150us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:27.762833 (+ 84us) write_op.cc:446] Released schema lock
0502 08:06:27.762875 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:27.763139 (+ 264us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:27.763162 (+ 23us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":109989,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":211,"apply.run_cpu_time_us":1758,"apply.run_wall_time_us":1758,"num_ops":2,"prepare.queue_time_us":7128,"prepare.run_cpu_time_us":3322,"prepare.run_wall_time_us":47459,"replication_time_us":151906}]]}
I20240502 08:06:28.038305 26396 catalog_manager.cc:5484] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:28.044281 26492 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:28.042559 26396 catalog_manager.cc:5484] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 4 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:28.051578 26541 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.054180 26541 leader_election.cc:302] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.064103 26541 leader_election.cc:302] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.065891 26541 leader_election.cc:302] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.066248 28087 raft_consensus.cc:2726] T 39e50c6374394235996663b59b4be48a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:28.067144 28087 raft_consensus.cc:2726] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 08:06:28.067849 28087 raft_consensus.cc:2726] T 24dcc168bb1b4064b7e7801a4f94f5ff P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:06:28.069329 28014 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:28.073781 28097 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240502 08:06:28.071982 28079 raft_consensus.cc:2726] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:06:28.072687 26642 raft_consensus.cc:1261] T 14320b2cc936478c8fe96bc8361ad980 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:28.081499 28097 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:28.086023 28097 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:28.079804 26541 leader_election.cc:302] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dc9560dbbe4d1aa44b2b2edda4080f; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:28.089224 28079 raft_consensus.cc:2726] T 00f1f297bff34c27b4789483f0b3668f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 08:06:28.055742 28069 raft_consensus.cc:2726] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:28.128487 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4429915312704856895d00171a6e6151"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:28.129225 26644 raft_consensus.cc:595] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:28.146781 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 62252.5us
I20240502 08:06:28.149210 27887 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:28.149647 27905 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:06:28.133934 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 126545us
I20240502 08:06:28.097494 28079 consensus_queue.cc:1035] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240502 08:06:28.105427 28014 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
I20240502 08:06:28.178663 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:28.179530 26642 raft_consensus.cc:595] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
W20240502 08:06:28.191716 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 179796us
I20240502 08:06:28.162688 28087 consensus_queue.cc:1035] T 14320b2cc936478c8fe96bc8361ad980 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20240502 08:06:28.282773 26575 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240502 08:06:28.335613 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6808b161f6ae43fda539a49cf1b811c4"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:28.336539 26644 raft_consensus.cc:595] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.339607 26644 raft_consensus.cc:603] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 08:06:28.292279 28081 raft_consensus.cc:2726] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:06:28.379453 28094 log.cc:924] Time spent T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20240502 08:06:28.379665 28096 log.cc:924] Time spent T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20240502 08:06:28.436306 28116 log.cc:924] Time spent T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240502 08:06:28.451326 26575 raft_consensus.cc:1261] T 9fba760137cc473c9fefe88a2cc261f0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:06:28.455121 28101 log.cc:924] Time spent T 5e0cb939b8c5477abcbd7707edbdc28b P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.288s user 0.001s sys 0.000s
W20240502 08:06:28.506419 28108 log.cc:924] Time spent T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
W20240502 08:06:28.508177 28110 log.cc:924] Time spent T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240502 08:06:28.482758 27978 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20240502 08:06:28.449414 28098 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
W20240502 08:06:28.565730 28103 log.cc:924] Time spent T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.290s user 0.000s sys 0.001s
W20240502 08:06:28.609103 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 45342.3us
W20240502 08:06:28.614100 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 69808.8us
W20240502 08:06:28.614145 28097 consensus_meta.cc:287] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa: Time spent flushing consensus metadata: real 0.527s user 0.013s sys 0.000s
I20240502 08:06:28.636147 26644 raft_consensus.cc:1261] T 9fba760137cc473c9fefe88a2cc261f0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:28.628558 28097 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:28.640920 28097 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:28.641498 26492 raft_consensus.cc:1226] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 5. Current term is 6. Ops: []
I20240502 08:06:28.605422 27824 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:28.646665 27983 consensus_queue.cc:1035] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:28.648010 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:28.655333 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:28.658855 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.663031 28097 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 08:06:28.751760 26644 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240502 08:06:28.767854 28102 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:28.788244 27178 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:28.804587 28105 log.cc:924] Time spent T 1d753d3fcb9249c28f415af2ab615999 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.380s user 0.000s sys 0.003s
I20240502 08:06:28.810793 27750 raft_consensus.cc:979] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:28.811815 27750 raft_consensus.cc:1067] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:28.814247 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca82734882074cecbdd750a509db67a0"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:28.814853 26642 raft_consensus.cc:595] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.816722 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 08:06:28.837522 28126 raft_consensus.cc:979] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:28.838106 28126 raft_consensus.cc:1067] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:28.848038 26567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:28.848640 26567 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:28.848706 26568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca82734882074cecbdd750a509db67a0"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:28.849047 26567 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:28.849319 26568 raft_consensus.cc:482] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:28.849848 26568 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:28.870254 26568 raft_consensus.cc:504] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:28.878882 26568 leader_election.cc:288] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:28.893851 26567 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:28.872112 28084 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:28.896891 26567 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:28.902437 27824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55494 (request call id 141) took 1304 ms. Trace:
I20240502 08:06:28.910144 28084 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:28.935758 27934 raft_consensus.cc:482] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:28.938951 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:28.939556 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:28.960168 26644 raft_consensus.cc:3032] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:28.960304 26642 raft_consensus.cc:3032] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:28.945406 28129 consensus_queue.cc:1046] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:28.960698 26644 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:28.960791 26642 raft_consensus.cc:729] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:28.964053 26642 consensus_queue.cc:260] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:28.965626 26644 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:28.967137 26644 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:28.971835 26642 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:28.991139 28114 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.556s user 0.004s sys 0.003s
I20240502 08:06:28.994836 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:28.996335 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:28.998229 26467 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa; no voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:28.968487 27934 raft_consensus.cc:504] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:29.015322 28084 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:29.020820 27934 leader_election.cc:288] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
W20240502 08:06:29.022308 28106 log.cc:924] Time spent T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.597s user 0.005s sys 0.004s
I20240502 08:06:29.023824 26499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60744 (request call id 559) took 1007 ms. Trace:
I20240502 08:06:29.024821 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:29.028900 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:29.030655 28081 raft_consensus.cc:2726] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 08:06:29.034683 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5216c4a3a33f47ac88fa35704487f77e" is_pre_election: true
I20240502 08:06:29.024330 26499 rpcz_store.cc:276] 0502 08:06:28.016841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:28.017072 (+ 231us) service_pool.cc:225] Handling call
0502 08:06:28.017446 (+ 374us) raft_consensus.cc:1457] Updating replica for 1 ops
0502 08:06:28.033116 (+ 15670us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:06:28.033146 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0502 08:06:28.056257 (+ 23111us) log.cc:842] Serialized 26 byte log entry
0502 08:06:28.190757 (+134500us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:06:28.430108 (+239351us) log.cc:842] Serialized 14 byte log entry
0502 08:06:28.430317 (+ 209us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:06:28.430381 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:06:29.021554 (+591173us) spinlock_profiling.cc:243] Waited 8.63 ms on lock 0x7b44000232cc. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f9701038b51 00007f9701106685 00007f97011070c2 00007f9703daf441 00007f9703daad4a 00007f9703d9eb1e 00007f9703d9de66 00007f9703d9da88 00007f9703da7481 00007f9703da7439 00007f9703da73c9
0502 08:06:29.023444 (+ 1890us) raft_consensus.cc:1671] finished
0502 08:06:29.023598 (+ 154us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:06:29.023753 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113595,"prepare.queue_time_us":236056,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":156,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":5187,"spinlock_wait_cycles":19857280,"thread_start_us":183737,"threads_started":2,"wal-append.queue_time_us":199209}
I20240502 08:06:29.042095 26615 leader_election.cc:302] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64; no voters: 5216c4a3a33f47ac88fa35704487f77e, 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:29.044978 26642 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:29.047375 26542 leader_election.cc:302] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:29.047075 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:06:29.060397 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 187255us
I20240502 08:06:29.060710 26644 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:29.061933 26542 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:29.063328 26495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca82734882074cecbdd750a509db67a0" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:29.063867 26495 raft_consensus.cc:3037] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:29.066648 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:29.067317 26499 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:29.073292 26495 raft_consensus.cc:2454] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:29.079349 26567 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240502 08:06:29.058557 28087 raft_consensus.cc:2781] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:29.084128 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab970a1718349829f0f76fa212d5594" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:29.090394 26499 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:29.079506 28124 raft_consensus.cc:2726] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240502 08:06:29.084986 28145 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:28.956473 27824 rpcz_store.cc:276] 0502 08:06:27.583251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:27.583420 (+ 169us) service_pool.cc:225] Handling call
0502 08:06:27.612615 (+ 29195us) log.cc:842] Serialized 494 byte log entry
0502 08:06:27.726131 (+113516us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b540042caa0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97040aa9b1 00007f97040b5470 00007f97040b5418 00007f97040b5398 00007f97040b5353 00007f97040b45dc 00007f970a124fd6 00007f970a11d5d8 00007f97040f15db 00007f97040f40d0
0502 08:06:27.763160 (+ 37029us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b540042caa0. stack: 00007f97011bfba3 00007f97011bfaa0 00007f9700b24584 00007f970a83a9e4 00007f970a83a979 00007f97040a16b1 00007f97040adc9f 00007f9704096795 00007f970409b9b1 00007f970409b939 00007f970409b8c9 00007f970409b891 00007f970409aa5d 00007f970a466da4 00007f970a466bd9 00007f97011f2236
0502 08:06:28.887826 (+1124666us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6687,"raft.queue_time_us":18071,"raft.run_cpu_time_us":3027,"raft.run_wall_time_us":162945,"spinlock_wait_cycles":11265920,"thread_start_us":31714,"threads_started":1,"wal-append.queue_time_us":32004,"wal-append.run_cpu_time_us":4338,"wal-append.run_wall_time_us":1093680}
I20240502 08:06:29.140581 27934 raft_consensus.cc:3032] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:29.141018 27934 raft_consensus.cc:729] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:29.141842 27934 consensus_queue.cc:260] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:29.143296 27934 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:29.146051 28145 raft_consensus.cc:686] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:29.147058 28145 consensus_queue.cc:237] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:29.168397 28138 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.246s user 0.000s sys 0.005s
I20240502 08:06:29.175619 26494 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240502 08:06:29.197094 28124 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:29.213208 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc069ce0343a4e9b9056d3604d2928df with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 08:06:29.219409 28146 log.cc:924] Time spent T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e: Append to log took a long time: real 0.120s user 0.004s sys 0.000s
I20240502 08:06:29.242434 26398 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:29.070084 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:29.250876 28087 raft_consensus.cc:686] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
W20240502 08:06:29.251170 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 165792us
W20240502 08:06:29.266031 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 133690us
I20240502 08:06:29.283087 28087 consensus_queue.cc:237] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:29.286975 28124 raft_consensus.cc:979] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:29.287603 28124 raft_consensus.cc:1067] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:29.307853 28129 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:29.329068 26396 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39272 (request call id 154) took 3274 ms. Trace:
I20240502 08:06:29.333638 26574 raft_consensus.cc:2932] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:29.329720 26396 rpcz_store.cc:276] 0502 08:06:26.054525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:26.054652 (+ 127us) service_pool.cc:225] Handling call
0502 08:06:29.329011 (+3274359us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:28.063373 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:28.063834 (+ 461us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:28.063878 (+ 44us) write_op.cc:427] Acquired schema lock
0502 08:06:28.063895 (+ 17us) tablet.cc:660] Decoding operations
0502 08:06:28.064604 (+ 709us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:28.064658 (+ 54us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:28.064684 (+ 26us) tablet.cc:683] Acquiring locks for 6 operations
0502 08:06:28.065270 (+ 586us) tablet.cc:699] Row locks acquired
0502 08:06:28.065298 (+ 28us) write_op.cc:260] PREPARE: finished
0502 08:06:28.065479 (+ 181us) write_op.cc:270] Start()
0502 08:06:28.065615 (+ 136us) write_op.cc:276] Timestamp: P: 1714637188065423 usec, L: 0
0502 08:06:28.065644 (+ 29us) op_driver.cc:331] REPLICATION: starting
0502 08:06:28.069078 (+ 3434us) log.cc:842] Serialized 2463 byte log entry
0502 08:06:28.088311 (+ 19233us) op_driver.cc:444] REPLICATION: finished
0502 08:06:28.091043 (+ 2732us) write_op.cc:301] APPLY: starting
0502 08:06:28.091122 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:28.091289 (+ 167us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:28.091386 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:28.092003 (+ 617us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:28.092203 (+ 200us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0502 08:06:28.092271 (+ 68us) write_op.cc:312] APPLY: finished
0502 08:06:28.106809 (+ 14538us) log.cc:842] Serialized 52 byte log entry
0502 08:06:28.130696 (+ 23887us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:28.136084 (+ 5388us) write_op.cc:446] Released schema lock
0502 08:06:28.137958 (+ 1874us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:28.177961 (+ 40003us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:28.183942 (+ 5981us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":5601,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":2510,"apply.run_cpu_time_us":3096,"apply.run_wall_time_us":115442,"num_ops":6,"prepare.queue_time_us":133,"prepare.run_cpu_time_us":3405,"prepare.run_wall_time_us":13413,"replication_time_us":12168}]]}
I20240502 08:06:29.337061 26399 catalog_manager.cc:5484] T 6aba3e3cbd94435eabac55bb87284652 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.351372 26567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4429915312704856895d00171a6e6151"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:29.351956 26567 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:29.352334 26567 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:29.355994 26399 catalog_manager.cc:5484] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.357852 26399 catalog_manager.cc:5484] T 147d10cad3be4380a85513c7cbaef19e P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.364223 26567 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:29.377924 26399 catalog_manager.cc:5484] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 2 to 3, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.380017 26399 catalog_manager.cc:5484] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.391659 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:29.392505 26493 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:29.394951 26567 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:29.399355 26399 catalog_manager.cc:5484] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 4, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.403009 26399 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 5 to 6, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 6 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.410959 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:29.411684 26641 raft_consensus.cc:3032] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:29.411995 26641 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:29.412277 26493 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:29.412834 26641 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:29.419694 26541 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:29.420982 28087 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:29.421029 26641 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:29.421661 28087 raft_consensus.cc:686] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:29.407790 26399 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 1 to 4, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.426999 26399 catalog_manager.cc:5484] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 3 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:29.447571 28087 consensus_queue.cc:237] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:29.504427 26641 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:29.616062 26395 catalog_manager.cc:5484] T 14320b2cc936478c8fe96bc8361ad980 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 2 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:29.635214 26395 catalog_manager.cc:5484] T 9fba760137cc473c9fefe88a2cc261f0 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 2 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 3 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:29.646391 26399 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39274 (request call id 140) took 1720 ms. Trace:
I20240502 08:06:29.646840 26399 rpcz_store.cc:276] 0502 08:06:27.925970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:06:27.926090 (+ 120us) service_pool.cc:225] Handling call
0502 08:06:29.646325 (+1720235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:06:29.482166 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:06:29.485293 (+ 3127us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:06:29.485325 (+ 32us) write_op.cc:427] Acquired schema lock
0502 08:06:29.485344 (+ 19us) tablet.cc:660] Decoding operations
0502 08:06:29.486348 (+ 1004us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:06:29.486404 (+ 56us) write_op.cc:633] Partition lock acquired for write op
0502 08:06:29.486454 (+ 50us) tablet.cc:683] Acquiring locks for 9 operations
0502 08:06:29.487220 (+ 766us) tablet.cc:699] Row locks acquired
0502 08:06:29.487237 (+ 17us) write_op.cc:260] PREPARE: finished
0502 08:06:29.487382 (+ 145us) write_op.cc:270] Start()
0502 08:06:29.487481 (+ 99us) write_op.cc:276] Timestamp: P: 1714637189487338 usec, L: 0
0502 08:06:29.487504 (+ 23us) op_driver.cc:331] REPLICATION: starting
0502 08:06:29.488102 (+ 598us) log.cc:842] Serialized 3614 byte log entry
0502 08:06:29.491266 (+ 3164us) op_driver.cc:444] REPLICATION: finished
0502 08:06:29.534069 (+ 42803us) write_op.cc:301] APPLY: starting
0502 08:06:29.534149 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0502 08:06:29.534410 (+ 261us) tablet.cc:1368] finished BulkCheckPresence
0502 08:06:29.534523 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:06:29.535061 (+ 538us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:06:29.535198 (+ 137us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0502 08:06:29.535226 (+ 28us) write_op.cc:312] APPLY: finished
0502 08:06:29.535422 (+ 196us) log.cc:842] Serialized 70 byte log entry
0502 08:06:29.535611 (+ 189us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:29.535719 (+ 108us) write_op.cc:446] Released schema lock
0502 08:06:29.535760 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0502 08:06:29.563721 (+ 27961us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:06:29.563755 (+ 34us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":58692,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":42058,"apply.run_cpu_time_us":2330,"apply.run_wall_time_us":30102,"num_ops":9,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":3707,"prepare.run_wall_time_us":9930,"replication_time_us":3674,"spinlock_wait_cycles":769280,"thread_start_us":38906,"threads_started":1}]]}
I20240502 08:06:29.658629 26645 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:29.660249 28087 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:29.700953 26499 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:29.700742 26397 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:29.705951 28145 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240502 08:06:29.741447 26491 tablet_service.cc:1514] Processing DeleteTablet for tablet fc069ce0343a4e9b9056d3604d2928df with delete_type TABLET_DATA_TOMBSTONED (TS 8f9fe1f3763547388c4301b13a127aaa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60736
I20240502 08:06:29.760660 26501 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:29.765628 28145 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:29.792423 26396 catalog_manager.cc:5484] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 4 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:29.793926 26396 catalog_manager.cc:5484] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 4 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:29.796458 26396 catalog_manager.cc:5484] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:29.844904 28171 tablet_replica.cc:330] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: stopping tablet replica
I20240502 08:06:29.847169 28171 raft_consensus.cc:2229] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:29.848043 28171 raft_consensus.cc:2258] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:29.851359 28171 ts_tablet_manager.cc:1875] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:29.888356 26645 raft_consensus.cc:1261] T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:29.897624 26567 consensus_queue.cc:237] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:29.898355 28156 consensus_queue.cc:1035] T ca82734882074cecbdd750a509db67a0 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:29.912300 27178 consensus_queue.cc:237] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:29.917447 26641 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:29.928366 27177 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240502 08:06:29.929911 26641 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240502 08:06:29.930326 26499 raft_consensus.cc:1261] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240502 08:06:29.930974 26493 raft_consensus.cc:1261] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240502 08:06:29.932463 28174 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:06:29.937609 28174 consensus_queue.cc:1035] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 08:06:29.938356 28014 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:29.949926 28098 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 5216c4a3a33f47ac88fa35704487f77e [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:29.951844 28086 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20240502 08:06:29.969518 26493 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:29.983484 28087 consensus_queue.cc:1035] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:30.037592 26644 raft_consensus.cc:2932] T 3ab970a1718349829f0f76fa212d5594 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 3 to 7, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:30.056737 26499 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:30.062877 28079 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240502 08:06:30.097237 26385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ab970a1718349829f0f76fa212d5594 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:30.098289 28087 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 3 to 6, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:30.103461 26395 catalog_manager.cc:5484] T 3ab970a1718349829f0f76fa212d5594 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: config changed from index 3 to 7, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New cstate: current_term: 6 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:30.119328 27177 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 5216c4a3a33f47ac88fa35704487f77e [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 3 to 6, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:30.125250 28171 ts_tablet_manager.cc:1888] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240502 08:06:30.125831 28171 log.cc:1196] T fc069ce0343a4e9b9056d3604d2928df P 8f9fe1f3763547388c4301b13a127aaa: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root/wals/fc069ce0343a4e9b9056d3604d2928df
W20240502 08:06:30.149677 28177 log.cc:924] Time spent T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.057s user 0.006s sys 0.000s
W20240502 08:06:30.149755 28175 log.cc:924] Time spent T ca82734882074cecbdd750a509db67a0 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.090s user 0.008s sys 0.000s
I20240502 08:06:30.158218 26499 raft_consensus.cc:2932] T 8d53d3e99887443b8967129ddd219819 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 3 to 6, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:30.176234 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d53d3e99887443b8967129ddd219819 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:30.191268 26562 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab970a1718349829f0f76fa212d5594 with delete_type TABLET_DATA_TOMBSTONED (TS 97dc9560dbbe4d1aa44b2b2edda4080f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51122
I20240502 08:06:30.187789 26385 catalog_manager.cc:4830] TS 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): tablet fc069ce0343a4e9b9056d3604d2928df (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:30.198815 26399 catalog_manager.cc:5484] T 8d53d3e99887443b8967129ddd219819 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 3 to 6, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New cstate: current_term: 6 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:30.233588 26631 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d53d3e99887443b8967129ddd219819 with delete_type TABLET_DATA_TOMBSTONED (TS 755f35bc66a841f382d1c4df6c6f6b64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55494
I20240502 08:06:30.237396 28180 tablet_replica.cc:330] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: stopping tablet replica
I20240502 08:06:30.242931 28180 raft_consensus.cc:2229] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:30.243785 28180 raft_consensus.cc:2258] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:30.241221 28181 tablet_replica.cc:330] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: stopping tablet replica
I20240502 08:06:30.244935 28181 raft_consensus.cc:2229] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:30.245615 28181 raft_consensus.cc:2258] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:30.248225 28181 ts_tablet_manager.cc:1875] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:30.256728 27978 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:06:30.257297 27978 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.260128 27978 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:30.261422 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" is_pre_election: true
I20240502 08:06:30.261879 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:30.262503 26645 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:30.263458 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:30.264145 27978 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240502 08:06:30.264514 27978 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:06:30.264856 27978 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:30.274070 27978 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.276628 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:30.290428 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:30.291091 26645 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Advancing to term 7
W20240502 08:06:30.297322 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 52557.8us
I20240502 08:06:30.297897 26645 raft_consensus.cc:2454] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 7.
I20240502 08:06:30.299124 26467 leader_election.cc:302] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters: 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:30.299151 28180 ts_tablet_manager.cc:1875] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:30.308056 27978 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:30.308701 27978 raft_consensus.cc:2781] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Leader election won for term 7
I20240502 08:06:30.310461 27978 raft_consensus.cc:686] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [term 7 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:30.311311 27978 consensus_queue.cc:237] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.323684 28087 raft_consensus.cc:482] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755f35bc66a841f382d1c4df6c6f6b64)
I20240502 08:06:30.324281 28087 raft_consensus.cc:504] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.331723 28087 leader_election.cc:288] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:30.332755 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64" is_pre_election: true
I20240502 08:06:30.333585 26649 raft_consensus.cc:2354] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 6. Current term is 7.
I20240502 08:06:30.334785 26542 leader_election.cc:398] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 6. Current term is 7.
I20240502 08:06:30.335098 26542 leader_election.cc:401] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240502 08:06:30.335722 28079 raft_consensus.cc:3037] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 7
I20240502 08:06:30.343274 28079 raft_consensus.cc:2726] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473) with higher term. Message: Invalid argument: T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dc9560dbbe4d1aa44b2b2edda4080f for earlier term 6. Current term is 7.
W20240502 08:06:30.346612 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 47604.1us
I20240502 08:06:30.348774 28181 ts_tablet_manager.cc:1888] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.5
I20240502 08:06:30.351739 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477dd390aca4aeabf4f028e82984ae2" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8f9fe1f3763547388c4301b13a127aaa" is_pre_election: true
I20240502 08:06:30.354692 28180 ts_tablet_manager.cc:1888] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20240502 08:06:30.356837 28180 log.cc:1196] T 3ab970a1718349829f0f76fa212d5594 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root/wals/3ab970a1718349829f0f76fa212d5594
I20240502 08:06:30.358887 28181 log.cc:1196] T 8d53d3e99887443b8967129ddd219819 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/wals/8d53d3e99887443b8967129ddd219819
I20240502 08:06:30.361260 26386 catalog_manager.cc:4830] TS 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): tablet 3ab970a1718349829f0f76fa212d5594 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:30.357617 26399 catalog_manager.cc:5484] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 5 to 7, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 7 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:30.364367 26386 catalog_manager.cc:4830] TS 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): tablet 8d53d3e99887443b8967129ddd219819 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:30.603013 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:30.701892 26568 raft_consensus.cc:1261] T c477dd390aca4aeabf4f028e82984ae2 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20240502 08:06:30.703888 28182 consensus_queue.cc:1035] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:30.714720 26649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:30.715337 26649 raft_consensus.cc:595] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:30.732466 26649 raft_consensus.cc:1261] T c477dd390aca4aeabf4f028e82984ae2 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20240502 08:06:30.733940 27978 consensus_queue.cc:1035] T c477dd390aca4aeabf4f028e82984ae2 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:30.788357 26645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a00601c8043478f8523c59e70966d50"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:30.789053 26645 raft_consensus.cc:595] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:30.792366 26645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:30.792953 26645 raft_consensus.cc:595] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:30.796613 26645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:30.797192 26645 raft_consensus.cc:595] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:30.820158 26649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:30.820761 26649 raft_consensus.cc:595] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:30.822126 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 08:06:30.826670 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:30.829452 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:30.955704 27620 raft_consensus.cc:979] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:30.956251 27620 raft_consensus.cc:1067] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:30.962638 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:30.964602 26499 raft_consensus.cc:482] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:30.965878 26499 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:30.970734 26499 raft_consensus.cc:504] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.973105 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:30.973738 26574 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:30.974953 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021cd2b53b784089b3fc08aad322e294" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:30.975549 26645 raft_consensus.cc:3032] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 08:06:30.975828 26645 raft_consensus.cc:729] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:30.976512 26645 consensus_queue.cc:260] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.977586 26645 raft_consensus.cc:3037] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:30.980693 26574 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 7.
I20240502 08:06:30.981853 26467 leader_election.cc:302] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:30.982622 27978 raft_consensus.cc:2781] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Leader election won for term 7
I20240502 08:06:30.983103 27978 raft_consensus.cc:686] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [term 7 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:30.984050 27978 consensus_queue.cc:237] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:30.991634 26499 leader_election.cc:288] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:30.995324 26645 raft_consensus.cc:2454] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 7.
I20240502 08:06:30.998989 28112 raft_consensus.cc:979] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:30.999495 28112 raft_consensus.cc:1067] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.009124 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a00601c8043478f8523c59e70966d50"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:31.009907 26492 raft_consensus.cc:482] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:31.010457 26492 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:31.028431 26399 catalog_manager.cc:5484] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 6 to 7, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 7 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:31.017517 26492 raft_consensus.cc:504] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.047075 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:31.047734 26574 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:31.063910 26574 raft_consensus.cc:2454] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:31.064272 26492 leader_election.cc:288] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:31.065461 26467 leader_election.cc:302] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:31.066622 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a00601c8043478f8523c59e70966d50" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:31.067219 26645 raft_consensus.cc:3032] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:31.067499 26645 raft_consensus.cc:729] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:31.067781 27978 raft_consensus.cc:2781] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:31.068303 27978 raft_consensus.cc:686] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:31.068212 26645 consensus_queue.cc:260] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.069468 26645 raft_consensus.cc:3037] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:31.069255 27978 consensus_queue.cc:237] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.069716 28112 raft_consensus.cc:979] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.072124 28112 raft_consensus.cc:1067] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.075654 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:31.076414 26499 raft_consensus.cc:482] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:31.076858 26499 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.079695 26645 raft_consensus.cc:2454] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:31.085683 26499 raft_consensus.cc:504] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.087577 26499 leader_election.cc:288] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:31.088841 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:31.089187 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fbc2401954920998f3ba61e449afd" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:31.089608 26645 raft_consensus.cc:3032] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:31.089980 26645 raft_consensus.cc:729] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:31.090102 26574 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.099938 26645 consensus_queue.cc:260] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.100922 26645 raft_consensus.cc:3037] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.141296 26574 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:31.142710 26467 leader_election.cc:302] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:31.144153 27978 raft_consensus.cc:2781] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:31.144668 27978 raft_consensus.cc:686] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:31.144743 26645 raft_consensus.cc:2454] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:31.145562 27978 consensus_queue.cc:237] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.173591 28035 raft_consensus.cc:979] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.174273 28035 raft_consensus.cc:1067] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.177137 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:31.177680 26492 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:31.178022 26492 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:31.192435 26492 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.195644 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:31.196323 26574 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:31.196048 26492 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:31.196637 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:31.197365 26645 raft_consensus.cc:3032] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:31.197676 26645 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:31.198508 26645 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.199849 26645 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:31.203519 26574 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:31.204643 26467 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:31.205523 27978 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:31.206074 27978 raft_consensus.cc:686] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:31.206951 27978 consensus_queue.cc:237] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.235561 26645 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:31.240787 26399 catalog_manager.cc:5484] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:31.243144 26399 catalog_manager.cc:5484] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:31.247790 26399 catalog_manager.cc:5484] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 5 to 6, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 6 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:31.293967 28035 raft_consensus.cc:979] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.294513 28035 raft_consensus.cc:1067] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:31.296123 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:31.296725 26492 raft_consensus.cc:482] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:31.297060 26492 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.307408 26492 raft_consensus.cc:504] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.309371 26492 leader_election.cc:288] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:31.310467 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:31.311129 26574 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.311362 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80a7fc6eb5046b78755fc53fcf8f1db" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:31.311933 26645 raft_consensus.cc:3032] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:31.312208 26645 raft_consensus.cc:729] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:31.312876 26645 consensus_queue.cc:260] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.313930 26645 raft_consensus.cc:3037] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:31.320537 26645 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:31.321542 26467 leader_election.cc:302] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:31.322412 27978 raft_consensus.cc:2781] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:31.322870 27978 raft_consensus.cc:686] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:31.323722 27978 consensus_queue.cc:237] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:31.368252 26399 catalog_manager.cc:5484] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:31.402660 26574 raft_consensus.cc:2454] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:31.444249 26568 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:31.445694 27978 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:31.464608 26645 raft_consensus.cc:1261] T 021cd2b53b784089b3fc08aad322e294 P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:31.466760 27978 consensus_queue.cc:1035] T 021cd2b53b784089b3fc08aad322e294 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:31.493105 26645 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:31.496912 28182 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:06:31.566133 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 53760.5us
I20240502 08:06:31.569097 26568 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:31.592433 28186 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:31.595453 26644 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:31.596798 28186 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:31.657208 26568 consensus_queue.cc:237] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:31.667799 26649 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:31.694065 26649 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:06:31.694603 26541 consensus_peers.cc:485] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6808b161f6ae43fda539a49cf1b811c4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:31.695413 28086 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:31.709100 28206 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:31.712575 26568 raft_consensus.cc:1261] T f57fbc2401954920998f3ba61e449afd P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:31.736827 26499 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:31.751478 28086 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:31.762869 26644 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:31.792354 26572 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:31.794541 28216 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:31.802412 28206 consensus_queue.cc:1035] T f57fbc2401954920998f3ba61e449afd P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:31.804317 28206 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:31.809672 26572 consensus_queue.cc:237] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:31.831954 26399 catalog_manager.cc:5484] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:31.838483 26649 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:31.875070 26492 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:31.879009 28206 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:06:31.886538 26541 consensus_peers.cc:485] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e067615c3a4148d597769a9f8a757e38. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:31.889338 26649 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:31.891072 28086 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:31.915004 26499 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:31.919896 28086 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:31.929245 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:31.962599 26572 raft_consensus.cc:1261] T a80a7fc6eb5046b78755fc53fcf8f1db P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:31.991425 28086 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:31.993395 26645 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.022316 26645 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:32.042670 28186 consensus_queue.cc:1035] T a80a7fc6eb5046b78755fc53fcf8f1db P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:32.058493 26499 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.066905 26396 catalog_manager.cc:5484] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:32.097097 26574 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:32.098923 28112 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:32.110364 26499 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:32.111876 26645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66aad5aa3c99405abb36667908363887"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:32.112515 26645 raft_consensus.cc:595] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:32.117151 26615 consensus_peers.cc:485] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b3c527a35284f68b946ea4bef6182c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:32.123298 28191 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:32.142427 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99e8fc1030084a55a81726683707e539"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:32.143090 26644 raft_consensus.cc:595] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:32.156180 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:32.156912 26644 raft_consensus.cc:595] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:32.158819 28112 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.166231 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:32.166939 26644 raft_consensus.cc:595] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:32.172871 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:32.173450 26644 raft_consensus.cc:595] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:32.175033 26499 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.175326 26574 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.179312 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 08:06:32.180923 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:32.181531 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:32.187588 26644 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:32.194728 28228 raft_consensus.cc:979] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.195186 28228 raft_consensus.cc:1067] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.196645 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66aad5aa3c99405abb36667908363887"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:32.197155 26492 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:32.197479 26492 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.202886 26397 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:32.236150 26572 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:32.238539 27620 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:32.217466 26492 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.249734 26492 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:32.264729 26572 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.251430 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:32.257393 26494 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240502 08:06:32.275331 26615 consensus_peers.cc:485] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcad69f7daa74587b431486e4cad620f. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:06:32.414323 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 138937us
I20240502 08:06:32.255213 28112 raft_consensus.cc:979] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.263099 28191 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Committing config change with OpId 4.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.300976 28234 ts_tablet_manager.cc:902] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:32.252136 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:32.420001 26574 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.429885 26644 raft_consensus.cc:3032] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:32.430444 26644 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:32.433446 26644 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.434959 28112 raft_consensus.cc:1067] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.436228 26644 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.483870 26644 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:32.486984 26574 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:32.503051 26467 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:32.505313 28186 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:32.505983 28186 raft_consensus.cc:686] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:32.500598 26397 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.507498 28186 consensus_queue.cc:237] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.526718 27935 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:32.531255 27935 raft_consensus.cc:979] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.531822 27935 raft_consensus.cc:1067] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.578835 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99e8fc1030084a55a81726683707e539"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:32.579602 26499 raft_consensus.cc:482] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:32.630712 26499 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.653900 26397 catalog_manager.cc:5484] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:32.688517 26494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:32.689208 26494 raft_consensus.cc:482] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:32.689589 26494 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.738193 28250 ts_tablet_manager.cc:902] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:32.729979 26499 raft_consensus.cc:504] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.732339 28247 ts_tablet_manager.cc:902] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:32.733002 28234 tablet_copy_client.cc:308] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.2:40601
I20240502 08:06:32.742138 26586 tablet_copy_service.cc:140] P 97dc9560dbbe4d1aa44b2b2edda4080f: Received BeginTabletCopySession request for tablet 6808b161f6ae43fda539a49cf1b811c4 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:43852)
I20240502 08:06:32.742842 26586 tablet_copy_service.cc:161] P 97dc9560dbbe4d1aa44b2b2edda4080f: Beginning new tablet copy session on tablet 6808b161f6ae43fda539a49cf1b811c4 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:43852: session id = 5216c4a3a33f47ac88fa35704487f77e-6808b161f6ae43fda539a49cf1b811c4
I20240502 08:06:32.746446 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:32.747282 26568 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.751677 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e8fc1030084a55a81726683707e539" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:32.755720 26644 raft_consensus.cc:3032] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:32.756114 26644 raft_consensus.cc:729] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:32.756901 26644 consensus_queue.cc:260] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.758137 26644 raft_consensus.cc:3037] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.751580 26497 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Committing config change with OpId 4.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:32.749513 26499 leader_election.cc:288] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:32.778429 26586 tablet_copy_source_session.cc:205] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:32.784206 26644 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:32.818862 26568 raft_consensus.cc:2454] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:32.815121 26494 raft_consensus.cc:504] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.838619 26467 leader_election.cc:302] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:32.840327 26494 leader_election.cc:288] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:32.848176 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:32.848994 26568 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.865281 27935 raft_consensus.cc:979] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.865936 27935 raft_consensus.cc:1067] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.872280 28242 raft_consensus.cc:979] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.872948 28242 raft_consensus.cc:1067] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:32.847419 28186 raft_consensus.cc:2781] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:32.902005 28234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6808b161f6ae43fda539a49cf1b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:32.890722 28186 raft_consensus.cc:686] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:32.925117 28186 consensus_queue.cc:237] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.937829 26568 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:32.952417 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345a6e692404520a7f76e5e3a0af8b0" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:32.953342 26641 raft_consensus.cc:3032] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:32.953727 26641 raft_consensus.cc:729] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:32.954900 26641 consensus_queue.cc:260] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:32.957033 26641 raft_consensus.cc:3037] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:32.990581 26501 raft_consensus.cc:1226] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:32.993467 26467 leader_election.cc:302] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:33.029229 28250 tablet_copy_client.cc:308] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.3:40473
I20240502 08:06:33.026118 26501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:33.036358 26501 raft_consensus.cc:482] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:33.037242 26501 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:33.045145 26644 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:33.048710 26641 raft_consensus.cc:2454] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:33.050875 26660 tablet_copy_service.cc:140] P 755f35bc66a841f382d1c4df6c6f6b64: Received BeginTabletCopySession request for tablet 6b3c527a35284f68b946ea4bef6182c9 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:54230)
I20240502 08:06:33.051522 26660 tablet_copy_service.cc:161] P 755f35bc66a841f382d1c4df6c6f6b64: Beginning new tablet copy session on tablet 6b3c527a35284f68b946ea4bef6182c9 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:54230: session id = 5216c4a3a33f47ac88fa35704487f77e-6b3c527a35284f68b946ea4bef6182c9
I20240502 08:06:33.052098 28241 raft_consensus.cc:2781] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:33.052711 28241 raft_consensus.cc:686] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:33.053609 28241 consensus_queue.cc:237] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.061615 26660 tablet_copy_source_session.cc:205] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:06:33.048816 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 211691us
I20240502 08:06:33.046923 28236 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:33.076658 26493 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:33.077317 26493 raft_consensus.cc:482] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:33.077692 26493 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:33.085813 28265 ts_tablet_manager.cc:902] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:33.101197 26501 raft_consensus.cc:504] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.110802 28247 tablet_copy_client.cc:308] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.2:40601
I20240502 08:06:33.114902 26573 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:33.116923 26586 tablet_copy_service.cc:140] P 97dc9560dbbe4d1aa44b2b2edda4080f: Received BeginTabletCopySession request for tablet e067615c3a4148d597769a9f8a757e38 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:43852)
I20240502 08:06:33.117522 26586 tablet_copy_service.cc:161] P 97dc9560dbbe4d1aa44b2b2edda4080f: Beginning new tablet copy session on tablet e067615c3a4148d597769a9f8a757e38 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:43852: session id = 5216c4a3a33f47ac88fa35704487f77e-e067615c3a4148d597769a9f8a757e38
I20240502 08:06:33.120709 28237 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:33.130336 28265 tablet_copy_client.cc:308] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.3:40473
I20240502 08:06:33.117547 28250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3c527a35284f68b946ea4bef6182c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:33.136272 26493 raft_consensus.cc:504] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.139371 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:33.139981 26568 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:33.142493 26493 leader_election.cc:288] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:33.149569 26660 tablet_copy_service.cc:140] P 755f35bc66a841f382d1c4df6c6f6b64: Received BeginTabletCopySession request for tablet bcad69f7daa74587b431486e4cad620f from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:54230)
I20240502 08:06:33.150313 26660 tablet_copy_service.cc:161] P 755f35bc66a841f382d1c4df6c6f6b64: Beginning new tablet copy session on tablet bcad69f7daa74587b431486e4cad620f from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:54230: session id = 5216c4a3a33f47ac88fa35704487f77e-bcad69f7daa74587b431486e4cad620f
I20240502 08:06:33.151139 28234 tablet_copy_client.cc:783] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:33.151901 28234 tablet_copy_client.cc:647] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:33.156553 28234 tablet_copy_client.cc:523] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:33.161985 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80f2fb50ae948dfb8cefba0fe36bd7f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:33.162743 26645 raft_consensus.cc:3032] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:33.163112 26645 raft_consensus.cc:729] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:33.163985 26645 consensus_queue.cc:260] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.164958 26645 raft_consensus.cc:3037] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:33.172261 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:33.172873 26572 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:33.183450 26586 tablet_copy_source_session.cc:205] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:33.196861 28247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e067615c3a4148d597769a9f8a757e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:33.207281 26501 leader_election.cc:288] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:33.165261 26397 catalog_manager.cc:5484] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:33.234185 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:33.234818 26649 raft_consensus.cc:3032] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:33.235117 26649 raft_consensus.cc:729] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:33.242573 26649 consensus_queue.cc:260] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.259186 26649 raft_consensus.cc:3037] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:33.281659 26572 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:33.282968 26660 tablet_copy_source_session.cc:205] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:33.283401 26467 leader_election.cc:302] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:33.284770 28241 raft_consensus.cc:2781] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:33.285282 28241 raft_consensus.cc:686] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:33.286130 28241 consensus_queue.cc:237] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.290557 28265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcad69f7daa74587b431486e4cad620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:33.345036 26644 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:33.348477 28267 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:33.388528 26568 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:33.390894 26467 leader_election.cc:302] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:33.392112 26567 raft_consensus.cc:1226] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 1. Current term is 2. Ops: []
I20240502 08:06:33.396396 28241 raft_consensus.cc:2781] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:33.404253 28241 raft_consensus.cc:686] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:33.405354 28241 consensus_queue.cc:237] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:33.436935 28234 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:33.486223 26567 raft_consensus.cc:1261] T 99e8fc1030084a55a81726683707e539 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:33.489638 28237 consensus_queue.cc:1035] T 99e8fc1030084a55a81726683707e539 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:33.490868 26649 raft_consensus.cc:2454] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:33.493767 26645 raft_consensus.cc:2454] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 2.
I20240502 08:06:33.502140 26576 raft_consensus.cc:1261] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:33.520397 28247 tablet_copy_client.cc:783] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:33.526443 28247 tablet_copy_client.cc:647] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:33.530506 28267 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240502 08:06:33.537245 28247 tablet_copy_client.cc:523] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:33.562359 28247 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:33.562671 26642 raft_consensus.cc:1261] T a345a6e692404520a7f76e5e3a0af8b0 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:33.578544 28265 tablet_copy_client.cc:783] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:33.636308 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:33.640689 28250 tablet_copy_client.cc:783] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:33.641486 28250 tablet_copy_client.cc:647] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:33.626389 28265 tablet_copy_client.cc:647] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:33.658552 28250 tablet_copy_client.cc:523] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:33.658260 28265 tablet_copy_client.cc:523] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:33.680627 26396 catalog_manager.cc:5484] T a345a6e692404520a7f76e5e3a0af8b0 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:33.681646 28241 consensus_queue.cc:1035] T a345a6e692404520a7f76e5e3a0af8b0 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:33.682358 28250 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:33.677956 28265 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:33.754606 26399 catalog_manager.cc:5484] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:33.757100 26399 catalog_manager.cc:5484] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:33.838368 28234 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:33.839713 28234 tablet_bootstrap.cc:492] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:33.840965 28234 ts_tablet_manager.cc:1367] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.404s user 0.021s sys 0.013s
I20240502 08:06:33.852560 26568 raft_consensus.cc:1261] T d80f2fb50ae948dfb8cefba0fe36bd7f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:33.854080 28241 consensus_queue.cc:1035] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:33.855546 26645 raft_consensus.cc:1261] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:33.869483 28186 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:33.993844 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:33.995136 26644 raft_consensus.cc:595] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:33.844559 28234 raft_consensus.cc:348] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:33.996395 28234 raft_consensus.cc:729] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:33.998615 28234 consensus_queue.cc:260] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.009965 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:34.018677 28234 ts_tablet_manager.cc:1398] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.177s user 0.011s sys 0.002s
I20240502 08:06:34.028718 26586 tablet_copy_service.cc:342] P 97dc9560dbbe4d1aa44b2b2edda4080f: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6808b161f6ae43fda539a49cf1b811c4 received from {username='slave'} at 127.0.0.1:43852
I20240502 08:06:34.029181 26586 tablet_copy_service.cc:433] P 97dc9560dbbe4d1aa44b2b2edda4080f: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6808b161f6ae43fda539a49cf1b811c4 on tablet 6808b161f6ae43fda539a49cf1b811c4 with peer 5216c4a3a33f47ac88fa35704487f77e
W20240502 08:06:34.033190 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 135842us
I20240502 08:06:34.034240 26642 raft_consensus.cc:595] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:34.034827 26642 raft_consensus.cc:603] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 08:06:34.063571 26644 raft_consensus.cc:1261] T d80f2fb50ae948dfb8cefba0fe36bd7f P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:34.170742 26575 raft_consensus.cc:1261] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:34.180446 28247 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:34.219092 28205 consensus_queue.cc:1035] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240502 08:06:34.220613 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:34.221273 26642 raft_consensus.cc:595] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:34.168848 28186 consensus_queue.cc:1035] T d80f2fb50ae948dfb8cefba0fe36bd7f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20240502 08:06:34.237349 28250 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:34.237740 28247 tablet_bootstrap.cc:492] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:34.268081 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4bbea332580449bbd811888439ee24f"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:34.268955 26642 raft_consensus.cc:595] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
W20240502 08:06:34.281287 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 78983.6us
I20240502 08:06:34.298959 28260 raft_consensus.cc:979] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.307245 27620 raft_consensus.cc:979] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.307688 27620 raft_consensus.cc:1067] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.310973 26495 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:34.311512 26495 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:34.311833 26495 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.304579 26642 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:34.312438 26642 raft_consensus.cc:595] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:34.303161 28247 ts_tablet_manager.cc:1367] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.741s user 0.027s sys 0.018s
I20240502 08:06:34.328209 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 08:06:34.336159 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:34.331239 28250 tablet_bootstrap.cc:492] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:34.356777 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:34.368548 28191 raft_consensus.cc:1067] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.378079 26493 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:34.361377 26495 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.378719 26493 raft_consensus.cc:482] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:34.379382 26493 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.390256 26495 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:34.386677 28250 ts_tablet_manager.cc:1367] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.705s user 0.038s sys 0.029s
I20240502 08:06:34.403568 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:34.404781 26567 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.409425 28265 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:34.412837 28247 raft_consensus.cc:348] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.413762 28247 raft_consensus.cc:729] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:34.414455 28247 consensus_queue.cc:260] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.417141 28247 ts_tablet_manager.cc:1398] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.088s user 0.006s sys 0.000s
I20240502 08:06:34.420976 28250 raft_consensus.cc:348] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.421914 28250 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:34.422610 28250 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.428910 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:34.429529 26642 raft_consensus.cc:3032] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:34.429808 26642 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:34.430508 26642 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.431747 26642 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.432660 28265 tablet_bootstrap.cc:492] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:34.423326 27177 raft_consensus.cc:1203] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240502 08:06:34.438486 28265 ts_tablet_manager.cc:1367] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.761s user 0.033s sys 0.027s
W20240502 08:06:34.442473 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 52669.8us
I20240502 08:06:34.441669 28265 raft_consensus.cc:348] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.444697 28265 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:34.444989 26642 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:34.445446 28265 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:34.446744 27178 raft_consensus.cc:1203] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240502 08:06:34.425915 28250 ts_tablet_manager.cc:1398] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.027s user 0.001s sys 0.005s
I20240502 08:06:34.455644 26493 raft_consensus.cc:504] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.457864 26660 tablet_copy_service.cc:342] P 755f35bc66a841f382d1c4df6c6f6b64: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6b3c527a35284f68b946ea4bef6182c9 received from {username='slave'} at 127.0.0.1:54230
I20240502 08:06:34.458365 26660 tablet_copy_service.cc:433] P 755f35bc66a841f382d1c4df6c6f6b64: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6b3c527a35284f68b946ea4bef6182c9 on tablet 6b3c527a35284f68b946ea4bef6182c9 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:34.464908 26586 tablet_copy_service.cc:342] P 97dc9560dbbe4d1aa44b2b2edda4080f: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-e067615c3a4148d597769a9f8a757e38 received from {username='slave'} at 127.0.0.1:43852
I20240502 08:06:34.465344 26586 tablet_copy_service.cc:433] P 97dc9560dbbe4d1aa44b2b2edda4080f: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-e067615c3a4148d597769a9f8a757e38 on tablet e067615c3a4148d597769a9f8a757e38 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:34.465900 28260 raft_consensus.cc:979] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.466449 28260 raft_consensus.cc:1067] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:34.464912 26493 leader_election.cc:288] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:34.468194 26567 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:34.474408 26493 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4bbea332580449bbd811888439ee24f"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:34.475618 26493 raft_consensus.cc:482] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:34.477936 26493 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.476920 26467 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:34.490134 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:34.490743 26567 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.496023 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:34.496557 26642 raft_consensus.cc:3032] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:34.496776 26642 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:34.500478 26642 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.504981 26642 raft_consensus.cc:3037] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.487936 28297 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:34.502386 28265 ts_tablet_manager.cc:1398] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.064s user 0.003s sys 0.005s
I20240502 08:06:34.515398 28297 raft_consensus.cc:686] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:34.521267 26642 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:34.523595 26567 raft_consensus.cc:2454] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:34.521732 26493 raft_consensus.cc:504] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.525509 28297 consensus_queue.cc:237] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.527951 26467 leader_election.cc:302] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:34.535835 28251 raft_consensus.cc:979] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:34.536329 28251 raft_consensus.cc:1067] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:34.541457 28186 raft_consensus.cc:2781] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:34.541977 28186 raft_consensus.cc:686] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:34.542994 26568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:34.542838 28186 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.543506 26568 raft_consensus.cc:482] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:34.543872 26568 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.550345 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:34.550928 26567 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.552996 26493 leader_election.cc:288] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:34.559329 26660 tablet_copy_service.cc:342] P 755f35bc66a841f382d1c4df6c6f6b64: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-bcad69f7daa74587b431486e4cad620f received from {username='slave'} at 127.0.0.1:54230
I20240502 08:06:34.569897 26660 tablet_copy_service.cc:433] P 755f35bc66a841f382d1c4df6c6f6b64: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-bcad69f7daa74587b431486e4cad620f on tablet bcad69f7daa74587b431486e4cad620f with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:34.560186 27177 raft_consensus.cc:1203] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Deduplicated request from leader. Original: 4.2->[4.3-4.3] Dedup: 4.3->[]
I20240502 08:06:34.574594 26568 raft_consensus.cc:504] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:34.579317 26568 leader_election.cc:288] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Requested vote from peers 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473), 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711)
I20240502 08:06:34.567442 26567 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:34.582412 27178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
I20240502 08:06:34.583006 27178 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.595876 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc069ce0343a4e9b9056d3604d2928df" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:34.597446 26642 raft_consensus.cc:3032] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:34.597806 26642 raft_consensus.cc:729] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:34.598742 26642 consensus_queue.cc:260] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:34.602705 26642 raft_consensus.cc:3037] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:34.608873 27178 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:34.632493 27177 raft_consensus.cc:1226] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Rejecting Update request from peer 755f35bc66a841f382d1c4df6c6f6b64 for earlier term 3. Current term is 4. Ops: []
I20240502 08:06:34.653435 26541 leader_election.cc:302] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:34.654348 28293 raft_consensus.cc:2781] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:34.654834 28293 raft_consensus.cc:686] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:34.655762 28293 consensus_queue.cc:237] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:34.656474 26395 catalog_manager.cc:5484] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:34.664341 26645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bbea332580449bbd811888439ee24f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:34.666390 26645 raft_consensus.cc:3032] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:34.666803 26645 raft_consensus.cc:729] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:34.669787 26645 consensus_queue.cc:260] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.668359 26467 leader_election.cc:302] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:34.684360 26645 raft_consensus.cc:3037] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:34.682914 26397 catalog_manager.cc:5484] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 4 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:34.681231 28297 raft_consensus.cc:2781] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:34.690943 28297 raft_consensus.cc:686] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:34.691776 28297 consensus_queue.cc:237] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:34.694324 26642 raft_consensus.cc:2454] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 4.
I20240502 08:06:34.697511 26645 raft_consensus.cc:2454] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:34.720952 28252 raft_consensus.cc:1050] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:34.726967 28252 consensus_queue.cc:237] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:34.796409 27178 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:34.797796 26645 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:34.802132 26500 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:34.801978 28299 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:34.804555 28252 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20240502 08:06:34.813930 28299 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:34.880198 27177 raft_consensus.cc:1203] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240502 08:06:34.888558 28293 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:34.891698 28307 raft_consensus.cc:1050] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:34.902833 26500 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:34.909293 28307 consensus_queue.cc:237] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:34.916352 26645 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:34.943382 26395 catalog_manager.cc:5484] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:34.935015 27178 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:34.991138 26569 consensus_queue.cc:237] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:35.019459 26645 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:35.026763 28235 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:35.028259 28251 raft_consensus.cc:1050] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:35.030833 28251 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 1, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:35.041957 27177 raft_consensus.cc:1261] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:35.040019 26395 catalog_manager.cc:5484] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:35.046578 26395 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:35.050942 26645 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.053151 28286 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.052434 28235 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.063589 26645 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:35.066165 28307 consensus_queue.cc:1035] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:35.067806 27177 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.068866 26500 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.070279 28293 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.070586 28237 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.081379 27178 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.087985 26642 raft_consensus.cc:2932] T e067615c3a4148d597769a9f8a757e38 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.105203 26644 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 08:06:35.095502 28293 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20240502 08:06:35.110050 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e067615c3a4148d597769a9f8a757e38 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:35.112170 26575 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:35.114171 28235 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:35.114504 28186 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.122826 26575 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:35.134431 28237 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.147065 26575 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:35.170441 27177 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:35.156580 28297 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240502 08:06:35.179889 26575 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:35.184360 28301 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.188319 28253 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:35.214087 26395 catalog_manager.cc:5484] T e067615c3a4148d597769a9f8a757e38 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:35.223610 26645 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:35.262979 26645 raft_consensus.cc:1261] T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:35.289245 26500 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 08:06:35.248358 28305 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240502 08:06:35.331019 27180 raft_consensus.cc:1261] T fc069ce0343a4e9b9056d3604d2928df P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240502 08:06:35.332019 28335 log.cc:924] Time spent T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
W20240502 08:06:35.334136 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 83273.8us
W20240502 08:06:35.354615 28328 log.cc:924] Time spent T fc069ce0343a4e9b9056d3604d2928df P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20240502 08:06:35.290426 28332 consensus_queue.cc:1035] T f4bbea332580449bbd811888439ee24f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20240502 08:06:35.324119 28235 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.412154 28191 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:35.460047 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240502 08:06:35.523434 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 66794.8us
I20240502 08:06:35.541352 28340 raft_consensus.cc:1050] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
W20240502 08:06:35.577741 28339 consensus_queue.cc:1159] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8f9fe1f3763547388c4301b13a127aaa" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20240502 08:06:35.580832 28340 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:35.607333 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 220978us
I20240502 08:06:35.643656 26395 catalog_manager.cc:5484] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:35.675576 28362 consensus_queue.cc:1035] T fc069ce0343a4e9b9056d3604d2928df P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20240502 08:06:35.677543 27180 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:35.689157 28353 log.cc:924] Time spent T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240502 08:06:35.692611 26647 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:35.693213 26647 raft_consensus.cc:595] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:35.693096 28339 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
W20240502 08:06:35.705745 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 211440us
I20240502 08:06:35.704609 27177 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.712502 26501 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.717159 26647 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.731487 26649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:35.732167 26649 raft_consensus.cc:595] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:35.747579 27179 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 LEARNER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.751049 26571 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.762764 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:35.763625 26644 raft_consensus.cc:595] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:35.768621 28260 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240502 08:06:35.771862 28354 log.cc:924] Time spent T f4bbea332580449bbd811888439ee24f P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.137s user 0.004s sys 0.000s
I20240502 08:06:35.774505 26576 consensus_queue.cc:237] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:35.786458 26491 tablet_service.cc:1514] Processing DeleteTablet for tablet e067615c3a4148d597769a9f8a757e38 with delete_type TABLET_DATA_TOMBSTONED (TS 8f9fe1f3763547388c4301b13a127aaa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60736
I20240502 08:06:35.796509 28191 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.802387 27180 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:35.818434 26650 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:35.819047 26650 raft_consensus.cc:595] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:35.818648 28235 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:35.824432 26501 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:06:35.845721 28191 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:35.843232 26495 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.860301 26650 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:35.863379 26650 raft_consensus.cc:595] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:35.871659 28368 tablet_replica.cc:330] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: stopping tablet replica
I20240502 08:06:35.873279 28368 raft_consensus.cc:2229] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:35.874547 28368 raft_consensus.cc:2258] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:35.879187 26571 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.893705 28260 raft_consensus.cc:979] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:35.894470 28251 raft_consensus.cc:979] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:35.905584 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 08:06:35.912667 28251 raft_consensus.cc:1067] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:35.899204 28368 ts_tablet_manager.cc:1875] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:35.917570 28299 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.919930 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:35.920651 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:35.915889 28260 raft_consensus.cc:1067] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:35.931421 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:35.932538 26492 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:35.933189 26492 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:35.935209 28300 raft_consensus.cc:979] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:35.935783 28301 raft_consensus.cc:1067] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:35.937250 26572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:35.937808 26572 raft_consensus.cc:482] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:35.938215 26572 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:35.942637 26573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:35.943439 26573 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:35.943881 26573 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:35.957237 26499 raft_consensus.cc:1261] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:35.970613 26492 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:35.972421 26492 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:35.978438 26572 raft_consensus.cc:504] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:35.925611 27179 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:35.989259 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:35.989910 26647 raft_consensus.cc:3032] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:35.990217 26647 raft_consensus.cc:729] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:35.990942 26647 consensus_queue.cc:260] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:35.992084 26647 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:35.999362 26647 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:36.000592 26542 leader_election.cc:302] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:36.001569 28342 raft_consensus.cc:2781] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:35.982865 26573 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.002067 28342 raft_consensus.cc:686] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:36.003608 26572 leader_election.cc:288] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:36.004827 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:36.005772 26647 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:36.006083 26647 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:36.007527 26573 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:36.010672 28260 raft_consensus.cc:979] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:36.011139 28260 raft_consensus.cc:979] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:36.011435 28339 raft_consensus.cc:1067] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:36.011526 28260 raft_consensus.cc:1067] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:36.008447 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:36.012159 28342 consensus_queue.cc:237] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.012710 26650 raft_consensus.cc:3032] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:36.013125 26650 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:36.013475 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:36.014003 26499 raft_consensus.cc:482] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:36.014422 26499 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.016899 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6808b161f6ae43fda539a49cf1b811c4 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:36.016981 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:36.017218 26492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:36.017580 26572 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:36.017746 26492 raft_consensus.cc:482] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:36.018299 26492 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:36.014011 26650 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.026436 26650 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.035897 26499 raft_consensus.cc:504] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.049908 26572 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:36.055377 26499 leader_election.cc:288] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:36.059393 28350 consensus_queue.cc:1035] T 6808b161f6ae43fda539a49cf1b811c4 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:36.059688 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c12a25498a426290c1ea595c9ac021" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:36.060335 26496 raft_consensus.cc:3037] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.032600 26395 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.066406 26467 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:36.066650 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:36.067277 26573 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.067791 26492 raft_consensus.cc:504] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.074260 28368 ts_tablet_manager.cc:1888] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240502 08:06:36.067235 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:36.085009 26492 leader_election.cc:288] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:36.082636 26399 catalog_manager.cc:5484] T 6808b161f6ae43fda539a49cf1b811c4 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.084620 28368 log.cc:1196] T e067615c3a4148d597769a9f8a757e38 P 8f9fe1f3763547388c4301b13a127aaa: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root/wals/e067615c3a4148d597769a9f8a757e38
I20240502 08:06:36.061676 26647 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.092263 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:36.092849 26572 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:36.088119 26496 raft_consensus.cc:2454] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:36.096765 26499 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.103282 28359 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:36.103823 28359 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:36.110925 26647 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:36.114445 28359 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.119087 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fa1083c65a4f999b1b52753bce70ed" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:36.119681 26649 raft_consensus.cc:3032] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:36.096406 28339 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.119973 26649 raft_consensus.cc:729] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:36.120851 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9686fb37d748508274be7bfb62c56f" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:36.121431 26644 raft_consensus.cc:3032] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 08:06:36.121711 26644 raft_consensus.cc:729] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:36.122460 26644 consensus_queue.cc:260] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.124884 26649 consensus_queue.cc:260] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.126206 26649 raft_consensus.cc:3037] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:36.123804 26644 raft_consensus.cc:3037] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:36.128877 27180 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.142777 26495 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.145100 26385 catalog_manager.cc:4830] TS 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): tablet e067615c3a4148d597769a9f8a757e38 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:36.154493 26631 tablet_service.cc:1514] Processing DeleteTablet for tablet 6808b161f6ae43fda539a49cf1b811c4 with delete_type TABLET_DATA_TOMBSTONED (TS 755f35bc66a841f382d1c4df6c6f6b64 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55494
I20240502 08:06:36.113587 26573 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:36.166774 26650 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:36.170480 26649 raft_consensus.cc:2454] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:36.175014 26647 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:36.164853 26467 leader_election.cc:302] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:36.164288 26493 raft_consensus.cc:2932] T 6808b161f6ae43fda539a49cf1b811c4 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.175791 26542 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:36.166667 26499 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:36.171937 26572 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 7.
I20240502 08:06:36.187980 28299 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:36.188522 28299 raft_consensus.cc:686] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:36.188951 26644 raft_consensus.cc:2454] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 7.
I20240502 08:06:36.189389 28299 consensus_queue.cc:237] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.205082 26399 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.210001 26467 leader_election.cc:302] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:36.224560 26647 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:36.232745 28358 raft_consensus.cc:2781] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:36.232982 28373 tablet_replica.cc:330] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: stopping tablet replica
I20240502 08:06:36.241514 28186 raft_consensus.cc:2781] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Leader election won for term 7
I20240502 08:06:36.239691 26397 catalog_manager.cc:5484] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:36.255661 27180 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 08:06:36.242516 28358 raft_consensus.cc:686] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:36.260819 26573 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 08:06:36.248041 28186 raft_consensus.cc:686] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [term 7 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:36.268908 28186 consensus_queue.cc:237] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.273017 28358 consensus_queue.cc:237] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:36.294354 28373 raft_consensus.cc:2229] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:36.296968 26399 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 5 to 6, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 6 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:36.302289 28340 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:36.302686 28373 raft_consensus.cc:2258] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 08:06:36.308257 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 45341.8us
I20240502 08:06:36.312157 26573 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.312966 28300 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20240502 08:06:36.320398 28373 ts_tablet_manager.cc:1875] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:36.330436 28340 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.336778 26572 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.340608 27180 raft_consensus.cc:2932] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.358593 26399 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 4 to 5, VOTER 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) evicted. New cstate: current_term: 4 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.373198 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bcad69f7daa74587b431486e4cad620f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:36.384279 26642 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:36.395718 26396 catalog_manager.cc:5484] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:36.404156 28373 ts_tablet_manager.cc:1888] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240502 08:06:36.401288 26397 catalog_manager.cc:5484] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:36.397737 26396 catalog_manager.cc:5484] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 6 to 7, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 7 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:36.422197 27179 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:36.423439 26499 raft_consensus.cc:1261] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 08:06:36.426605 28300 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 08:06:36.429312 26491 tablet_service.cc:1514] Processing DeleteTablet for tablet bcad69f7daa74587b431486e4cad620f with delete_type TABLET_DATA_TOMBSTONED (TS 8f9fe1f3763547388c4301b13a127aaa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60736
I20240502 08:06:36.430289 28368 tablet_replica.cc:330] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: stopping tablet replica
I20240502 08:06:36.431208 28368 raft_consensus.cc:2229] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:36.431936 28368 raft_consensus.cc:2258] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:36.425263 28373 log.cc:1196] T 6808b161f6ae43fda539a49cf1b811c4 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/wals/6808b161f6ae43fda539a49cf1b811c4
I20240502 08:06:36.436180 28368 ts_tablet_manager.cc:1875] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:36.466328 26647 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.433434 28340 consensus_queue.cc:1035] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20240502 08:06:36.446658 26386 catalog_manager.cc:4830] TS 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): tablet 6808b161f6ae43fda539a49cf1b811c4 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:36.470980 28339 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.473228 28350 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:36.480683 26499 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.489178 26386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b3c527a35284f68b946ea4bef6182c9 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:36.492007 27180 raft_consensus.cc:2932] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.514369 26397 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: config changed from index 4 to 5, VOTER 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) evicted. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:36.525691 26562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3c527a35284f68b946ea4bef6182c9 with delete_type TABLET_DATA_TOMBSTONED (TS 97dc9560dbbe4d1aa44b2b2edda4080f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51122
I20240502 08:06:36.534811 26573 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:36.537587 28358 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:36.618361 26499 raft_consensus.cc:1261] T a3c12a25498a426290c1ea595c9ac021 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.618775 28378 tablet_replica.cc:330] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: stopping tablet replica
I20240502 08:06:36.639463 28378 raft_consensus.cc:2229] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:36.640825 28378 raft_consensus.cc:2258] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:36.646608 28342 consensus_queue.cc:1035] T a3c12a25498a426290c1ea595c9ac021 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:36.648160 28378 ts_tablet_manager.cc:1875] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:06:36.702548 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 45823.8us
I20240502 08:06:36.701001 28368 ts_tablet_manager.cc:1888] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.4
I20240502 08:06:36.711360 28368 log.cc:1196] T bcad69f7daa74587b431486e4cad620f P 8f9fe1f3763547388c4301b13a127aaa: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-0-root/wals/bcad69f7daa74587b431486e4cad620f
I20240502 08:06:36.715826 26385 catalog_manager.cc:4830] TS 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577): tablet bcad69f7daa74587b431486e4cad620f (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:36.718220 26647 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240502 08:06:36.720232 28186 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:06:36.766521 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 61398.6us
I20240502 08:06:36.774359 26492 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.790323 28324 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240502 08:06:36.822559 26573 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.849861 26644 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.853767 28342 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:36.855554 26642 raft_consensus.cc:1261] T b5fa1083c65a4f999b1b52753bce70ed P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:36.861311 26570 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:36.862342 28345 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:36.866709 28345 consensus_queue.cc:1035] T b5fa1083c65a4f999b1b52753bce70ed P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:36.868099 28186 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:36.870954 26647 raft_consensus.cc:1261] T fa9686fb37d748508274be7bfb62c56f P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:36.902303 28391 consensus_queue.cc:1035] T fa9686fb37d748508274be7bfb62c56f P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:36.978133 28378 ts_tablet_manager.cc:1888] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240502 08:06:36.978654 28378 log.cc:1196] T 6b3c527a35284f68b946ea4bef6182c9 P 97dc9560dbbe4d1aa44b2b2edda4080f: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-1-root/wals/6b3c527a35284f68b946ea4bef6182c9
W20240502 08:06:36.992803 28392 log.cc:924] Time spent T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20240502 08:06:37.006774 26386 catalog_manager.cc:4830] TS 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601): tablet 6b3c527a35284f68b946ea4bef6182c9 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:37.057776 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:37.220238 26647 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:37.220875 26647 raft_consensus.cc:595] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:37.248127 26647 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:37.248867 26647 raft_consensus.cc:595] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:37.263311 28340 raft_consensus.cc:979] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:37.268427 28340 raft_consensus.cc:1067] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:37.263821 26496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:37.274310 26496 raft_consensus.cc:595] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:37.276329 26570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:37.276975 26570 raft_consensus.cc:482] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.277352 26570 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.326674 26570 raft_consensus.cc:504] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.332151 26570 leader_election.cc:288] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:37.343777 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:37.344228 26496 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.352327 26496 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:37.355767 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbfc58321f544229109ce9344e7000d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:37.356282 26647 raft_consensus.cc:3032] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:37.358717 28391 raft_consensus.cc:979] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:37.359251 28391 raft_consensus.cc:1067] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:37.359849 26541 leader_election.cc:302] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:37.357108 26647 raft_consensus.cc:729] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:37.360771 26644 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55500
I20240502 08:06:37.361291 28235 raft_consensus.cc:2781] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:37.361400 26644 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.361858 28235 raft_consensus.cc:686] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:37.361896 26644 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:37.361403 26647 consensus_queue.cc:260] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.363808 28235 consensus_queue.cc:237] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.364358 26647 raft_consensus.cc:3037] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.370607 26644 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.373281 26647 raft_consensus.cc:2454] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:37.376560 26644 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:37.378597 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:37.379187 26568 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:37.378186 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:37.379741 26496 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Stepping down as leader of term 6
I20240502 08:06:37.380050 26496 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:37.380728 26496 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.381780 26496 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:37.392076 26568 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 7.
I20240502 08:06:37.394027 26616 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:37.394865 28349 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 08:06:37.395344 28349 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:37.396209 26496 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 7.
I20240502 08:06:37.396143 28349 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.408504 26396 catalog_manager.cc:5484] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:37.411255 26397 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 6 to 7, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 7 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:37.418571 26573 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:37.419227 26573 raft_consensus.cc:595] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:37.421890 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "5216c4a3a33f47ac88fa35704487f77e"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:37.422530 26644 raft_consensus.cc:595] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:37.425961 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:37.426869 26644 raft_consensus.cc:595] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:37.427929 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240502 08:06:37.429770 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:37.430491 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:37.593791 28339 raft_consensus.cc:979] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:37.594357 28340 raft_consensus.cc:1067] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:37.599768 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60752
I20240502 08:06:37.600349 26499 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.600708 26499 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:37.756139 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 44727.5us
I20240502 08:06:37.744510 26499 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.787176 26499 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:37.795055 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:37.795931 26573 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:37.800098 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:37.800750 26647 raft_consensus.cc:3032] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:37.801030 26647 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:37.801750 26647 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.802785 26647 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:37.804569 26573 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:37.809067 26647 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:37.812476 26467 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:37.824332 28410 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:37.824862 28410 raft_consensus.cc:686] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:37.850237 28410 consensus_queue.cc:237] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.875257 26496 raft_consensus.cc:1261] T 6dbfc58321f544229109ce9344e7000d P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:37.890434 26647 raft_consensus.cc:1261] T 6dbfc58321f544229109ce9344e7000d P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:37.898300 28340 raft_consensus.cc:979] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 5216c4a3a33f47ac88fa35704487f77e to start an election
I20240502 08:06:37.898862 28340 raft_consensus.cc:1067] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 5216c4a3a33f47ac88fa35704487f77e to start an election
I20240502 08:06:37.902922 28419 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:37.902662 26492 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:37.915771 28349 raft_consensus.cc:979] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:37.917685 28349 raft_consensus.cc:1067] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:37.919059 28235 consensus_queue.cc:1035] T 6dbfc58321f544229109ce9344e7000d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:37.919499 28409 raft_consensus.cc:979] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:37.919946 28409 raft_consensus.cc:1067] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:37.921391 26572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:37.921948 26572 raft_consensus.cc:482] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.922456 26572 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.922430 26496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60744
I20240502 08:06:37.923118 26496 raft_consensus.cc:482] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.923467 26496 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.929575 26496 raft_consensus.cc:504] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.933667 26572 raft_consensus.cc:504] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.937669 26572 leader_election.cc:288] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:37.940440 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:37.941653 26493 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.941303 26397 catalog_manager.cc:5484] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:37.951871 27180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f"
dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
from {username='slave'} at 127.0.0.1:38552
I20240502 08:06:37.952520 27180 raft_consensus.cc:482] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:37.952940 27180 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.956279 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:37.959134 26572 raft_consensus.cc:3032] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:37.960947 26572 raft_consensus.cc:729] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:37.962304 26572 consensus_queue.cc:260] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.963526 26572 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.964406 27180 raft_consensus.cc:504] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:37.964705 26493 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:37.970708 26541 leader_election.cc:302] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:37.974421 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ed3fca6dd84c87bc322d121bdeff7b" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:37.970818 26496 leader_election.cc:288] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:37.976172 26644 raft_consensus.cc:3037] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.977579 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfba0fadba745b3ad79a60a2faccabb" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:37.978353 26647 raft_consensus.cc:3032] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:37.978628 26647 raft_consensus.cc:729] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:37.980801 26572 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:37.987051 26647 consensus_queue.cc:260] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:37.988469 26647 raft_consensus.cc:3037] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:37.989960 26467 leader_election.cc:302] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:37.997251 26496 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:38.006150 26644 raft_consensus.cc:2454] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:38.006156 28408 raft_consensus.cc:2781] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:38.011312 28359 raft_consensus.cc:2781] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:38.011312 28408 raft_consensus.cc:686] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:38.012480 28408 consensus_queue.cc:237] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:38.018072 26647 raft_consensus.cc:2454] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:38.024330 26572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:38.025113 26572 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:38.031913 26573 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:38.019510 28359 raft_consensus.cc:686] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:38.036635 26572 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 5.
I20240502 08:06:38.038288 27145 leader_election.cc:302] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:38.039196 28343 raft_consensus.cc:2781] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:38.039685 28343 raft_consensus.cc:686] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [term 5 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:38.040552 28343 consensus_queue.cc:237] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:38.041431 28359 consensus_queue.cc:237] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:38.019721 28414 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240502 08:06:38.053197 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcad69f7daa74587b431486e4cad620f" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:38.053933 26649 raft_consensus.cc:3032] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:38.054229 26649 raft_consensus.cc:729] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:38.054550 26644 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 08:06:38.054889 26649 consensus_queue.cc:260] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:38.055951 26649 raft_consensus.cc:3037] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:38.042922 27180 leader_election.cc:288] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:38.073758 26649 raft_consensus.cc:2454] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 5.
I20240502 08:06:38.061312 28391 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240502 08:06:38.059921 28407 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240502 08:06:38.098482 26572 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 08:06:38.100363 28412 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:38.099792 26395 catalog_manager.cc:5484] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4). New cstate: current_term: 5 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:06:38.110234 26466 consensus_peers.cc:485] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7ac53bd5d944d0bbc8062cd0ba0550a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:38.138667 26396 catalog_manager.cc:5484] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:38.143419 28358 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:38.150368 26647 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:38.146720 26572 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
W20240502 08:06:38.205761 26447 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 755f35bc66a841f382d1c4df6c6f6b64 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } }
I20240502 08:06:38.212044 26399 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 5, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
W20240502 08:06:38.219483 26447 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4efaf1504b1842b3a948e6e10898efc5, TS 8f9fe1f3763547388c4301b13a127aaa -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:38.220077 26447 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4efaf1504b1842b3a948e6e10898efc5, TS 8f9fe1f3763547388c4301b13a127aaa -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:38.238916 26447 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ec54ed4ed93048d4a896c43796e5727d, TS 97dc9560dbbe4d1aa44b2b2edda4080f -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:38.239270 26447 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ec54ed4ed93048d4a896c43796e5727d, TS 97dc9560dbbe4d1aa44b2b2edda4080f -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:38.296986 26447 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f57fbc2401954920998f3ba61e449afd, TS 755f35bc66a841f382d1c4df6c6f6b64 -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:38.297363 26447 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f57fbc2401954920998f3ba61e449afd, TS 755f35bc66a841f382d1c4df6c6f6b64 -> TS 5216c4a3a33f47ac88fa35704487f77e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:38.305367 26397 catalog_manager.cc:5484] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:38.342501 26572 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:38.353324 28358 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:38.476220 26644 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:38.487562 26492 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:38.504688 28342 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:38.510582 26572 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240502 08:06:38.514569 28343 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:38.522768 28359 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:38.536721 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:38.549688 26644 raft_consensus.cc:1261] T fcfba0fadba745b3ad79a60a2faccabb P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:38.553746 28408 consensus_queue.cc:1035] T fcfba0fadba745b3ad79a60a2faccabb P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:38.569808 26572 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:38.596637 26647 raft_consensus.cc:1261] T bcad69f7daa74587b431486e4cad620f P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 5216c4a3a33f47ac88fa35704487f77e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240502 08:06:38.603533 28411 consensus_queue.cc:1035] T bcad69f7daa74587b431486e4cad620f P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:38.712061 26647 raft_consensus.cc:1261] T 90ed3fca6dd84c87bc322d121bdeff7b P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:38.754670 28359 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:38.756110 28448 ts_tablet_manager.cc:902] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:38.799597 28391 consensus_queue.cc:1035] T 90ed3fca6dd84c87bc322d121bdeff7b P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:38.869994 26650 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:38.870640 26650 raft_consensus.cc:595] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:38.878199 26573 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:38.879001 26573 raft_consensus.cc:595] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:38.897718 26492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:38.898391 26492 raft_consensus.cc:595] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:38.916563 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 08:06:38.919436 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:38.924131 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:38.933645 28448 tablet_copy_client.cc:308] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.1:40577
I20240502 08:06:38.939498 26511 tablet_copy_service.cc:140] P 8f9fe1f3763547388c4301b13a127aaa: Received BeginTabletCopySession request for tablet e7ac53bd5d944d0bbc8062cd0ba0550a from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:58270)
I20240502 08:06:38.940124 26511 tablet_copy_service.cc:161] P 8f9fe1f3763547388c4301b13a127aaa: Beginning new tablet copy session on tablet e7ac53bd5d944d0bbc8062cd0ba0550a from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:58270: session id = 5216c4a3a33f47ac88fa35704487f77e-e7ac53bd5d944d0bbc8062cd0ba0550a
I20240502 08:06:38.951233 26511 tablet_copy_source_session.cc:205] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:38.960824 28448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ac53bd5d944d0bbc8062cd0ba0550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:38.987773 28450 raft_consensus.cc:979] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:38.988361 28391 raft_consensus.cc:1067] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:38.991465 26647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55500
I20240502 08:06:38.992025 26647 raft_consensus.cc:482] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:38.992367 26647 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:38.998433 26647 raft_consensus.cc:504] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.001230 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:39.001322 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54ed4ed93048d4a896c43796e5727d" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:39.001961 26573 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:39.001931 26492 raft_consensus.cc:3032] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Stepping down as leader of term 6
I20240502 08:06:39.003793 26492 raft_consensus.cc:729] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:39.004523 26492 consensus_queue.cc:260] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.006814 26492 raft_consensus.cc:3037] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Advancing to term 7
I20240502 08:06:39.015090 26492 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 7.
I20240502 08:06:39.016222 26615 leader_election.cc:302] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:39.017009 28406 raft_consensus.cc:2781] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 08:06:39.017491 28406 raft_consensus.cc:686] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:39.019228 28406 consensus_queue.cc:237] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.030383 28448 tablet_copy_client.cc:783] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:39.031008 28448 tablet_copy_client.cc:647] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:39.032546 26647 leader_election.cc:288] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 7 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:39.035091 26573 raft_consensus.cc:2454] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 7.
I20240502 08:06:39.037760 28428 raft_consensus.cc:979] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:39.038241 28428 raft_consensus.cc:1067] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:39.038935 28448 tablet_copy_client.cc:523] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:39.043354 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60744
I20240502 08:06:39.043922 26499 raft_consensus.cc:482] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:39.044292 26499 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:39.053195 26499 raft_consensus.cc:504] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.068665 26499 leader_election.cc:288] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:39.072597 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:39.073098 26650 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:39.074028 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb83d16acf49edbe7b8b774d7d7bca" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:39.074528 26573 raft_consensus.cc:3032] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:39.074734 26573 raft_consensus.cc:729] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:39.077368 26399 catalog_manager.cc:5484] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 6 to 7, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 7 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:39.080001 26650 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:39.084198 26467 leader_election.cc:302] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:39.084980 28391 raft_consensus.cc:2781] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:39.085460 28391 raft_consensus.cc:686] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [term 6 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:39.086236 28391 consensus_queue.cc:237] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.075273 26573 consensus_queue.cc:260] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.094453 26573 raft_consensus.cc:3037] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:39.104218 26573 raft_consensus.cc:2454] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 6.
I20240502 08:06:39.121013 28448 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:39.135516 26397 catalog_manager.cc:5484] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 5 to 6, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 6 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:39.246012 28339 raft_consensus.cc:979] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:39.246688 28339 raft_consensus.cc:1067] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:39.249203 26573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:39.249873 26573 raft_consensus.cc:482] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:39.250276 26573 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:39.273468 28448 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:39.283649 28448 tablet_bootstrap.cc:492] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:39.284998 28448 ts_tablet_manager.cc:1367] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.164s user 0.034s sys 0.001s
I20240502 08:06:39.288198 28448 raft_consensus.cc:348] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:39.289477 28448 raft_consensus.cc:729] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:39.290325 28448 consensus_queue.cc:260] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:39.297065 28448 ts_tablet_manager.cc:1398] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 08:06:39.307055 26511 tablet_copy_service.cc:342] P 8f9fe1f3763547388c4301b13a127aaa: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-e7ac53bd5d944d0bbc8062cd0ba0550a received from {username='slave'} at 127.0.0.1:58270
I20240502 08:06:39.307526 26511 tablet_copy_service.cc:433] P 8f9fe1f3763547388c4301b13a127aaa: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-e7ac53bd5d944d0bbc8062cd0ba0550a on tablet e7ac53bd5d944d0bbc8062cd0ba0550a with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:39.362955 26573 raft_consensus.cc:504] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.397609 26573 leader_election.cc:288] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:39.399492 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:39.400173 26644 raft_consensus.cc:3032] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:39.400501 26644 raft_consensus.cc:729] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:39.401237 26644 consensus_queue.cc:260] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.404825 26644 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:39.408463 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a8eccc1a164ecbb7d5ada16f48b6de" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:39.409085 26499 raft_consensus.cc:3037] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:39.411713 26644 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:39.412557 26542 leader_election.cc:302] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:39.414695 28324 raft_consensus.cc:2781] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:39.415210 28324 raft_consensus.cc:686] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:39.416028 28324 consensus_queue.cc:237] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:39.424302 26499 raft_consensus.cc:2454] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:39.426616 26492 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:39.430327 28406 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:39.468695 26397 catalog_manager.cc:5484] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:39.501581 26570 raft_consensus.cc:1261] T ec54ed4ed93048d4a896c43796e5727d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240502 08:06:39.512629 26570 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:39.513983 26650 raft_consensus.cc:1261] T b9fb83d16acf49edbe7b8b774d7d7bca P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:39.514251 28358 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:06:39.511307 28414 consensus_queue.cc:1035] T ec54ed4ed93048d4a896c43796e5727d P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240502 08:06:39.518117 28358 consensus_queue.cc:1035] T b9fb83d16acf49edbe7b8b774d7d7bca P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:39.601524 27180 raft_consensus.cc:1203] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20240502 08:06:39.826476 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 44021.5us
W20240502 08:06:39.829247 26615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 48153.1us
W20240502 08:06:39.850986 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 72675.7us
I20240502 08:06:39.898242 26649 raft_consensus.cc:1261] T a1a8eccc1a164ecbb7d5ada16f48b6de P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:39.899801 28408 consensus_queue.cc:1035] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:39.964430 26496 raft_consensus.cc:1261] T a1a8eccc1a164ecbb7d5ada16f48b6de P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:39.965950 28476 consensus_queue.cc:1035] T a1a8eccc1a164ecbb7d5ada16f48b6de P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:40.080438 28450 raft_consensus.cc:1050] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:40.083209 28450 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:40.091362 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:40.107373 26642 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 08:06:40.108050 27180 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEARNER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 08:06:40.109604 28390 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:40.132812 28391 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:40.133724 26568 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 08:06:40.138739 28391 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:40.165691 28450 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.179380 26642 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.182741 26568 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.196249 27180 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.202730 26399 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: config changed from index 5 to 6, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:40.295933 26499 consensus_queue.cc:237] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:40.307009 27180 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240502 08:06:40.308519 28450 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:06:40.316412 28359 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.318159 27180 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.323447 26573 raft_consensus.cc:1261] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240502 08:06:40.325244 28450 consensus_queue.cc:1035] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 08:06:40.335626 26385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7ac53bd5d944d0bbc8062cd0ba0550a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:40.338248 26573 raft_consensus.cc:2932] T e7ac53bd5d944d0bbc8062cd0ba0550a P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.360716 26397 catalog_manager.cc:5484] T e7ac53bd5d944d0bbc8062cd0ba0550a P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 6 to 7, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:40.387213 26647 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:40.402279 26647 raft_consensus.cc:595] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:40.406320 26570 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:40.406956 26570 raft_consensus.cc:595] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:40.414180 26499 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7ac53bd5d944d0bbc8062cd0ba0550a"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:40.414744 26499 raft_consensus.cc:595] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:40.415091 26499 raft_consensus.cc:619] T e7ac53bd5d944d0bbc8062cd0ba0550a P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Rejecting request to transfer leadership because tablet server 755f35bc66a841f382d1c4df6c6f6b64 is not a voter in the active config
I20240502 08:06:40.416704 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 08:06:40.418478 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:06:40.418861 27144 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.1:40577, user_credentials={real_user=slave}} blocked reactor thread for 79805us
I20240502 08:06:40.419175 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:40.452322 26631 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ac53bd5d944d0bbc8062cd0ba0550a with delete_type TABLET_DATA_TOMBSTONED (TS 755f35bc66a841f382d1c4df6c6f6b64 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55494
I20240502 08:06:40.458971 28406 raft_consensus.cc:979] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:40.459570 28339 raft_consensus.cc:1067] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:40.465567 26568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:40.466193 26568 raft_consensus.cc:482] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:40.466593 26568 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:40.478793 28491 tablet_replica.cc:330] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: stopping tablet replica
I20240502 08:06:40.479449 26568 raft_consensus.cc:504] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.480347 28491 raft_consensus.cc:2229] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:40.481122 28491 raft_consensus.cc:2258] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:40.482872 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:40.483530 26647 raft_consensus.cc:3032] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:06:40.483847 26647 raft_consensus.cc:729] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:40.484978 26647 consensus_queue.cc:260] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.486268 26647 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:40.486137 28491 ts_tablet_manager.cc:1875] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:40.493703 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0774e7aa819c48f9bcda1d69e4e88a79" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:40.494375 26499 raft_consensus.cc:3037] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:06:40.508764 26499 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:40.517248 26568 leader_election.cc:288] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:40.537159 28428 raft_consensus.cc:979] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:40.537667 28428 raft_consensus.cc:1067] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:40.538341 26541 leader_election.cc:302] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:40.540940 28477 raft_consensus.cc:2781] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:06:40.541452 28477 raft_consensus.cc:686] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:40.542416 28477 consensus_queue.cc:237] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.559981 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60744
I20240502 08:06:40.560717 26499 raft_consensus.cc:482] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:40.561210 26499 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:40.564163 28491 ts_tablet_manager.cc:1888] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20240502 08:06:40.567639 28491 log.cc:1196] T e7ac53bd5d944d0bbc8062cd0ba0550a P 755f35bc66a841f382d1c4df6c6f6b64: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/wals/e7ac53bd5d944d0bbc8062cd0ba0550a
I20240502 08:06:40.569828 26386 catalog_manager.cc:4830] TS 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): tablet e7ac53bd5d944d0bbc8062cd0ba0550a (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
I20240502 08:06:40.573199 26647 raft_consensus.cc:2454] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 2.
I20240502 08:06:40.578214 26499 raft_consensus.cc:504] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.582106 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:40.582703 26568 raft_consensus.cc:3032] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:40.582989 26568 raft_consensus.cc:729] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:40.583631 26568 consensus_queue.cc:260] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.584628 26568 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:40.598606 26499 leader_election.cc:288] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:40.598834 26568 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:40.601460 26397 catalog_manager.cc:5484] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 1 to 2, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 2 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:40.602811 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2d8b29e0844df3a3267e030a5aa644" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:40.603425 26649 raft_consensus.cc:3037] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:40.605916 26467 leader_election.cc:302] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:40.606743 28391 raft_consensus.cc:2781] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:40.607249 28391 raft_consensus.cc:686] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:40.608078 28391 consensus_queue.cc:237] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:40.632632 26649 raft_consensus.cc:2454] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 5.
I20240502 08:06:40.666329 26397 catalog_manager.cc:5484] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 4 to 5, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 5 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:41.055814 26647 raft_consensus.cc:1261] T 0774e7aa819c48f9bcda1d69e4e88a79 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:41.057348 28324 consensus_queue.cc:1035] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:06:41.134177 26499 raft_consensus.cc:1261] T 0774e7aa819c48f9bcda1d69e4e88a79 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:06:41.135790 28324 consensus_queue.cc:1035] T 0774e7aa819c48f9bcda1d69e4e88a79 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240502 08:06:41.181452 26573 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:41.183645 28450 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240502 08:06:41.242833 26647 raft_consensus.cc:1261] T 5b2d8b29e0844df3a3267e030a5aa644 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:41.276643 28391 consensus_queue.cc:1035] T 5b2d8b29e0844df3a3267e030a5aa644 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:41.559084 26649 consensus_queue.cc:237] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:41.596735 26499 raft_consensus.cc:1261] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:41.599682 28478 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:06:41.601912 26615 consensus_peers.cc:485] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33e1db2e4cc44834966a5a186294dff8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:41.603864 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:41.609143 26573 raft_consensus.cc:1261] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:41.614627 28406 consensus_queue.cc:1035] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:41.649976 28504 raft_consensus.cc:2932] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.657851 26492 raft_consensus.cc:2932] T 33e1db2e4cc44834966a5a186294dff8 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.668869 26496 consensus_queue.cc:237] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:41.683619 26397 catalog_manager.cc:5484] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:41.686266 26568 raft_consensus.cc:2932] T 33e1db2e4cc44834966a5a186294dff8 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.744177 26649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:41.744812 26649 raft_consensus.cc:595] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:41.768958 26568 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240502 08:06:41.770625 26466 consensus_peers.cc:485] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a00601c8043478f8523c59e70966d50. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:41.774564 26647 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:06:41.789103 28359 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:41.789981 28391 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:41.818524 26492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:41.819109 26492 raft_consensus.cc:595] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:41.838908 28391 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.868398 26568 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.896629 26649 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.932052 26649 consensus_queue.cc:237] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:41.952431 28505 raft_consensus.cc:979] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:41.952950 28505 raft_consensus.cc:1067] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:41.954562 26573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:41.955103 26573 raft_consensus.cc:482] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:41.955446 26573 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:41.957007 26496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66aad5aa3c99405abb36667908363887"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:41.957583 26496 raft_consensus.cc:595] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:41.963464 26399 catalog_manager.cc:5484] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:41.965386 26573 raft_consensus.cc:504] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:41.971793 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:41.971998 26573 leader_election.cc:288] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:41.972441 26647 raft_consensus.cc:3032] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:06:41.972767 26647 raft_consensus.cc:729] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:41.973457 26647 consensus_queue.cc:260] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:41.982630 26647 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:41.992983 26647 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:42.025502 26568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:42.126960 26568 raft_consensus.cc:595] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:42.140710 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
W20240502 08:06:42.142254 26615 consensus_peers.cc:485] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a6f8df55d8c4456ac8d84fdcb919e55. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:42.142570 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:42.143275 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:42.144436 26572 raft_consensus.cc:1261] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:42.155674 26496 raft_consensus.cc:1261] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:06:42.156008 26542 leader_election.cc:302] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:42.159294 28476 raft_consensus.cc:2781] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:06:42.159932 28476 raft_consensus.cc:686] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:42.161748 28476 consensus_queue.cc:237] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.165231 28414 consensus_queue.cc:1035] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:06:42.200716 28524 ts_tablet_manager.cc:902] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:42.216137 28504 consensus_queue.cc:1035] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 08:06:42.220147 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 83502.1us
I20240502 08:06:42.221598 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b344ca7b5ec4def84739ae6b81fa851" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:42.222275 26499 raft_consensus.cc:3037] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:06:42.233088 26499 raft_consensus.cc:2454] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 3.
I20240502 08:06:42.239849 28406 raft_consensus.cc:2932] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:42.241745 28516 raft_consensus.cc:979] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
W20240502 08:06:42.221830 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 76068.8us
I20240502 08:06:42.243546 26496 raft_consensus.cc:2932] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 8f9fe1f3763547388c4301b13a127aaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:42.262248 28359 raft_consensus.cc:1067] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:42.264467 28501 raft_consensus.cc:979] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:42.284886 28359 raft_consensus.cc:1067] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:42.313894 26395 catalog_manager.cc:5484] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 2 to 3, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 3 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:42.354390 28535 ts_tablet_manager.cc:902] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577)
I20240502 08:06:42.364588 26647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55500
I20240502 08:06:42.365445 26647 raft_consensus.cc:482] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:42.365840 26647 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.366312 26649 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66aad5aa3c99405abb36667908363887"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55500
I20240502 08:06:42.372638 26649 raft_consensus.cc:482] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:42.373159 26649 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.381968 26647 raft_consensus.cc:504] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.385298 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:42.385893 26493 raft_consensus.cc:3032] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:42.386731 26493 raft_consensus.cc:729] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:42.389022 26493 consensus_queue.cc:260] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.389998 26647 leader_election.cc:288] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:42.385787 26395 catalog_manager.cc:5484] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 1 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:42.394948 26493 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.396653 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efaf1504b1842b3a948e6e10898efc5" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:42.397755 26573 raft_consensus.cc:3037] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.406329 26649 raft_consensus.cc:504] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.408921 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:42.410161 26649 leader_election.cc:288] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:42.411307 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aad5aa3c99405abb36667908363887" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:42.411852 26571 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.417712 26573 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:42.423452 26616 leader_election.cc:302] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:42.424644 26571 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:42.417758 26493 raft_consensus.cc:2454] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
I20240502 08:06:42.426605 28324 raft_consensus.cc:979] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:42.427047 28324 raft_consensus.cc:1067] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:42.431164 26496 raft_consensus.cc:3032] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:42.431578 26496 raft_consensus.cc:729] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:42.432616 26496 consensus_queue.cc:260] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.457298 26501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60744
I20240502 08:06:42.457984 26501 raft_consensus.cc:482] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:42.458388 26501 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:42.467612 26501 raft_consensus.cc:504] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.471976 26573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:42.472780 26573 raft_consensus.cc:3032] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:42.473158 26573 raft_consensus.cc:729] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:42.474464 26501 leader_election.cc:288] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:42.474067 26573 consensus_queue.cc:260] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.476388 26573 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:42.479605 26496 raft_consensus.cc:3037] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:42.496121 26496 raft_consensus.cc:2454] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 5.
W20240502 08:06:42.496341 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 52224.3us
I20240502 08:06:42.496598 26576 raft_consensus.cc:1226] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Rejecting Update request from peer 8f9fe1f3763547388c4301b13a127aaa for earlier term 4. Current term is 5. Ops: []
I20240502 08:06:42.497421 26616 leader_election.cc:302] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:42.466948 28534 raft_consensus.cc:2781] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
W20240502 08:06:42.503509 27145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 68249.2us
I20240502 08:06:42.506872 26573 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:42.508909 26649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc16fa0dfe47728a16df1a9fd99c84" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:42.509531 26649 raft_consensus.cc:3037] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:42.526681 28505 raft_consensus.cc:2781] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:42.527393 28505 raft_consensus.cc:686] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:42.517213 26647 consensus_queue.cc:237] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:42.528571 28505 consensus_queue.cc:237] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.531530 26649 raft_consensus.cc:2454] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:42.532698 26571 raft_consensus.cc:2932] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:42.542526 28534 raft_consensus.cc:686] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:42.550823 28524 tablet_copy_client.cc:308] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.3:40473
I20240502 08:06:42.550841 28534 consensus_queue.cc:237] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:42.634531 28546 log.cc:924] Time spent T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20240502 08:06:42.559644 28535 tablet_copy_client.cc:308] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.1:40577
I20240502 08:06:42.617033 26660 tablet_copy_service.cc:140] P 755f35bc66a841f382d1c4df6c6f6b64: Received BeginTabletCopySession request for tablet 33e1db2e4cc44834966a5a186294dff8 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:54230)
I20240502 08:06:42.651122 26660 tablet_copy_service.cc:161] P 755f35bc66a841f382d1c4df6c6f6b64: Beginning new tablet copy session on tablet 33e1db2e4cc44834966a5a186294dff8 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:54230: session id = 5216c4a3a33f47ac88fa35704487f77e-33e1db2e4cc44834966a5a186294dff8
I20240502 08:06:42.654798 26511 tablet_copy_service.cc:140] P 8f9fe1f3763547388c4301b13a127aaa: Received BeginTabletCopySession request for tablet 6a00601c8043478f8523c59e70966d50 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:58270)
I20240502 08:06:42.655975 26511 tablet_copy_service.cc:161] P 8f9fe1f3763547388c4301b13a127aaa: Beginning new tablet copy session on tablet 6a00601c8043478f8523c59e70966d50 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:58270: session id = 5216c4a3a33f47ac88fa35704487f77e-6a00601c8043478f8523c59e70966d50
W20240502 08:06:42.612998 26616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 76819.9us
I20240502 08:06:42.585768 26467 leader_election.cc:302] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9fe1f3763547388c4301b13a127aaa, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
W20240502 08:06:42.660751 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.194.2:40601, user_credentials={real_user=slave}} blocked reactor thread for 75539.7us
I20240502 08:06:42.662408 28530 raft_consensus.cc:2781] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:42.681161 26649 raft_consensus.cc:1261] T 0b344ca7b5ec4def84739ae6b81fa851 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:42.681216 28530 raft_consensus.cc:686] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:42.686693 28530 consensus_queue.cc:237] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:42.693462 26501 raft_consensus.cc:1261] T 0b344ca7b5ec4def84739ae6b81fa851 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:06:42.702582 26615 consensus_peers.cc:485] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5145fc92dff54e38b92911a9a14b978e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:06:42.709395 26568 raft_consensus.cc:1261] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:42.709729 26493 raft_consensus.cc:1261] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:06:42.723150 28521 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:42.726189 28504 consensus_queue.cc:1035] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:42.737529 26660 tablet_copy_source_session.cc:205] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:42.737555 28324 consensus_queue.cc:1035] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240502 08:06:42.772598 28524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e1db2e4cc44834966a5a186294dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:42.788627 26511 tablet_copy_source_session.cc:205] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:42.790354 26395 catalog_manager.cc:5484] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 4 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:42.810000 28535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a00601c8043478f8523c59e70966d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:06:42.847008 28549 consensus_queue.cc:1035] T 0b344ca7b5ec4def84739ae6b81fa851 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:42.857347 28551 ts_tablet_manager.cc:902] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:42.865991 26496 raft_consensus.cc:2932] T 5145fc92dff54e38b92911a9a14b978e P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:42.873943 28524 tablet_copy_client.cc:783] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:42.874706 28524 tablet_copy_client.cc:647] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:42.880125 28535 tablet_copy_client.cc:783] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:42.880851 28535 tablet_copy_client.cc:647] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:42.862758 28521 raft_consensus.cc:2932] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:42.896911 26399 catalog_manager.cc:5484] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:42.899822 28524 tablet_copy_client.cc:523] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:42.906967 26395 catalog_manager.cc:5484] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 4 to 5, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 5 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
I20240502 08:06:42.903760 28535 tablet_copy_client.cc:523] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:42.968851 28524 tablet_bootstrap.cc:492] T 33e1db2e4cc44834966a5a186294dff8 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:42.969517 26395 catalog_manager.cc:5484] T 5145fc92dff54e38b92911a9a14b978e P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New cstate: current_term: 3 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:42.998364 26571 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:43.000660 28504 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:43.030684 26568 raft_consensus.cc:2932] T 5145fc92dff54e38b92911a9a14b978e P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } attrs { replace: true } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } } }
I20240502 08:06:43.038290 26499 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:43.039880 28519 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:43.063637 28535 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
I20240502 08:06:43.087373 26496 raft_consensus.cc:1261] T 66aad5aa3c99405abb36667908363887 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:43.102799 28414 consensus_queue.cc:1035] T 66aad5aa3c99405abb36667908363887 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240502 08:06:43.108533 26573 raft_consensus.cc:1261] T 4efaf1504b1842b3a948e6e10898efc5 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240502 08:06:43.124413 28519 consensus_queue.cc:1035] T 4efaf1504b1842b3a948e6e10898efc5 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20240502 08:06:43.253988 26647 raft_consensus.cc:1261] T 72cc16fa0dfe47728a16df1a9fd99c84 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:43.266556 28450 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:06:43.277987 26571 raft_consensus.cc:1261] T 72cc16fa0dfe47728a16df1a9fd99c84 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240502 08:06:43.279593 28551 tablet_copy_client.cc:308] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.194.3:40473
I20240502 08:06:43.281126 28530 consensus_queue.cc:1035] T 72cc16fa0dfe47728a16df1a9fd99c84 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:06:43.282732 26660 tablet_copy_service.cc:140] P 755f35bc66a841f382d1c4df6c6f6b64: Received BeginTabletCopySession request for tablet 2a6f8df55d8c4456ac8d84fdcb919e55 from peer 5216c4a3a33f47ac88fa35704487f77e ({username='slave'} at 127.0.0.1:54230)
I20240502 08:06:43.283282 26660 tablet_copy_service.cc:161] P 755f35bc66a841f382d1c4df6c6f6b64: Beginning new tablet copy session on tablet 2a6f8df55d8c4456ac8d84fdcb919e55 from peer 5216c4a3a33f47ac88fa35704487f77e at {username='slave'} at 127.0.0.1:54230: session id = 5216c4a3a33f47ac88fa35704487f77e-2a6f8df55d8c4456ac8d84fdcb919e55
I20240502 08:06:43.318478 28572 ts_tablet_manager.cc:902] T 5145fc92dff54e38b92911a9a14b978e P 5216c4a3a33f47ac88fa35704487f77e: Initiating tablet copy from peer 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
F20240502 08:06:43.319356 28572 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.queue_time_us, tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.queue_time_us) (tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.run_cpu_time_us, tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.run_cpu_time_us) (tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.run_wall_time_us, tablet-download-pool-2a6f8df55d8c4456ac8d84fdcb919e55.run_wall_time_us) (tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.queue_time_us, tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.queue_time_us) (tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.run_cpu_time_us, tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.run_cpu_time_us) (tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.run_wall_time_us, tablet-download-pool-33e1db2e4cc44834966a5a186294dff8.run_wall_time_us) (tablet-download-pool-3ab970a1718349829f0f76fa212d5594.queue_time_us, tablet-download-pool-3ab970a1718349829f0f76fa212d5594.queue_time_us) (tablet-download-pool-3ab970a1718349829f0f76fa212d5594.run_cpu_time_us, tablet-download-pool-3ab970a1718349829f0f76fa212d5594.run_cpu_time_us) (tablet-download-pool-3ab970a1718349829f0f76fa212d5594.run_wall_time_us, tablet-download-pool-3ab970a1718349829f0f76fa212d5594.run_wall_time_us) (tablet-download-pool-5145fc92dff54e38b92911a9a14b978e.queue_time_us, tablet-download-pool-5145fc92dff54e38b92911a9a14b978e.queue_time_us) (tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.queue_time_us, tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.queue_time_us) (tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.run_cpu_time_us, tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.run_cpu_time_us) (tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.run_wall_time_us, tablet-download-pool-6808b161f6ae43fda539a49cf1b811c4.run_wall_time_us) (tablet-download-pool-6a00601c8043478f8523c59e70966d50.queue_time_us, tablet-download-pool-6a00601c8043478f8523c59e70966d50.queue_time_us) (tablet-download-pool-6a00601c8043478f8523c59e70966d50.run_cpu_time_us, tablet-download-pool-6a00601c8043478f8523c59e70966d50.run_cpu_time_us) (tablet-download-pool-6a00601c8043478f8523c59e70966d50.run_wall_time_us, tablet-download-pool-6a00601c8043478f8523c59e70966d50.run_wall_time_us) (tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.queue_time_us, tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.queue_time_us) (tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.run_cpu_time_us, tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.run_cpu_time_us) (tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.run_wall_time_us, tablet-download-pool-6b3c527a35284f68b946ea4bef6182c9.run_wall_time_us) (tablet-download-pool-8d53d3e99887443b8967129ddd219819.queue_time_us, tablet-download-pool-8d53d3e99887443b8967129ddd219819.queue_time_us) (tablet-download-pool-8d53d3e99887443b8967129ddd219819.run_cpu_time_us, tablet-download-pool-8d53d3e99887443b8967129ddd219819.run_cpu_time_us) (tablet-download-pool-8d53d3e99887443b8967129ddd219819.run_wall_time_us, tablet-download-pool-8d53d3e99887443b8967129ddd219819.run_wall_time_us) (tablet-download-pool-9a26db3828484577a5da525732f0b102.queue_time_us, tablet-download-pool-9a26db3828484577a5da525732f0b102.queue_time_us) (tablet-download-pool-9a26db3828484577a5da525732f0b102.run_cpu_time_us, tablet-download-pool-9a26db3828484577a5da525732f0b102.run_cpu_time_us) (tablet-download-pool-9a26db3828484577a5da525732f0b102.run_wall_time_us, tablet-download-pool-9a26db3828484577a5da525732f0b102.run_wall_time_us) (tablet-download-pool-bcad69f7daa74587b431486e4cad620f.queue_time_us, tablet-download-pool-bcad69f7daa74587b431486e4cad620f.queue_time_us) (tablet-download-pool-bcad69f7daa74587b431486e4cad620f.run_cpu_time_us, tablet-download-pool-bcad69f7daa74587b431486e4cad620f.run_cpu_time_us) (tablet-download-pool-bcad69f7daa74587b431486e4cad620f.run_wall_time_us, tablet-download-pool-bcad69f7daa74587b431486e4cad620f.run_wall_time_us) (tablet-download-pool-e067615c3a4148d597769a9f8a757e38.queue_time_us, tablet-download-pool-e067615c3a4148d597769a9f8a757e38.queue_time_us) (tablet-download-pool-e067615c3a4148d597769a9f8a757e38.run_cpu_time_us, tablet-download-pool-e067615c3a4148d597769a9f8a757e38.run_cpu_time_us) (tablet-download-pool-e067615c3a4148d597769a9f8a757e38.run_wall_time_us, tablet-download-pool-e067615c3a4148d597769a9f8a757e38.run_wall_time_us) (tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.queue_time_us, tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.queue_time_us) (tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.run_cpu_time_us, tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.run_cpu_time_us) (tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.run_wall_time_us, tablet-download-pool-e7ac53bd5d944d0bbc8062cd0ba0550a.run_wall_time_us) (tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.queue_time_us, tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.queue_time_us) (tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.run_cpu_time_us, tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.run_cpu_time_us) (tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.run_wall_time_us, tablet-download-pool-fc069ce0343a4e9b9056d3604d2928df.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 08:06:43.325452 26660 tablet_copy_source_session.cc:205] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:06:43.325860 28535 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:06:43.397373 28535 tablet_bootstrap.cc:492] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap complete.
I20240502 08:06:43.398195 28535 ts_tablet_manager.cc:1367] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Time spent bootstrapping tablet: real 0.335s user 0.025s sys 0.011s
I20240502 08:06:43.372560 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:06:43.399621 28551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6f8df55d8c4456ac8d84fdcb919e55. 1 dirs total, 0 dirs full, 0 dirs failed
*** Check failure stack trace: ***
*** Aborted at 1714637203 (unix time) try "date -d @1714637203" if you are using GNU date ***
I20240502 08:06:43.409260 28535 raft_consensus.cc:348] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:43.413347 28535 raft_consensus.cc:729] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Initialized, Role: LEARNER
I20240502 08:06:43.415958 28535 consensus_queue.cc:260] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: NON_VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: true } }
I20240502 08:06:43.513602 27179 raft_consensus.cc:1203] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 08:06:43.514138 28551 tablet_copy_client.cc:783] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 0 data blocks...
I20240502 08:06:43.518070 28535 ts_tablet_manager.cc:1398] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e: Time spent starting tablet: real 0.119s user 0.007s sys 0.000s
I20240502 08:06:43.520085 26511 tablet_copy_service.cc:342] P 8f9fe1f3763547388c4301b13a127aaa: Request end of tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6a00601c8043478f8523c59e70966d50 received from {username='slave'} at 127.0.0.1:58270
I20240502 08:06:43.520669 26511 tablet_copy_service.cc:433] P 8f9fe1f3763547388c4301b13a127aaa: ending tablet copy session 5216c4a3a33f47ac88fa35704487f77e-6a00601c8043478f8523c59e70966d50 on tablet 6a00601c8043478f8523c59e70966d50 with peer 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:43.523677 28551 tablet_copy_client.cc:647] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:06:43.604614 28551 tablet_copy_client.cc:523] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:06:43.616570 28551 tablet_bootstrap.cc:492] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 5216c4a3a33f47ac88fa35704487f77e: Bootstrap starting.
W20240502 08:06:43.620569 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 53776.4us
W20240502 08:06:43.648479 26541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.4:45711, user_credentials={real_user=slave}} blocked reactor thread for 80547.6us
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006308) received by PID 25352 (TID 0x7f962bc72700) from PID 25352; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240502 08:06:43.808638 26571 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4429915312704856895d00171a6e6151"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:43.809448 26571 raft_consensus.cc:595] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
I20240502 08:06:43.827010 26650 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:43.827617 26650 raft_consensus.cc:595] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:43.839563 26496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:43.840206 26496 raft_consensus.cc:595] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:43.842785 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 08:06:43.893026 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:43.899020 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
@ 0x7f9704522980 (unknown) at ??:0
I20240502 08:06:43.934291 28521 raft_consensus.cc:979] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:43.934790 28504 raft_consensus.cc:1067] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:43.938470 26571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:43.939251 26571 raft_consensus.cc:482] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:43.939713 26571 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 7 FOLLOWER]: Advancing to term 8
I20240502 08:06:43.961087 26571 raft_consensus.cc:504] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:43.972832 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:43.973405 26571 leader_election.cc:288] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 8 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:43.973583 26496 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 7 FOLLOWER]: Advancing to term 8
I20240502 08:06:43.977732 26647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f68e2a58df64f808deb0f0d3e63e43d" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:43.978875 26647 raft_consensus.cc:3032] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 08:06:43.979159 26647 raft_consensus.cc:729] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:43.979913 26647 consensus_queue.cc:260] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:43.981231 26647 raft_consensus.cc:3037] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 7 FOLLOWER]: Advancing to term 8
I20240502 08:06:43.998270 26647 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 8.
I20240502 08:06:44.000751 26542 leader_election.cc:302] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:44.001449 28530 raft_consensus.cc:1050] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa: attempting to promote NON_VOTER 5216c4a3a33f47ac88fa35704487f77e to VOTER
I20240502 08:06:44.001905 28522 raft_consensus.cc:2781] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 8 FOLLOWER]: Leader election won for term 8
I20240502 08:06:44.002503 28522 raft_consensus.cc:686] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [term 8 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:44.003480 28522 consensus_queue.cc:237] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.004146 28530 consensus_queue.cc:237] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:43.992888 26496 raft_consensus.cc:2454] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 8.
I20240502 08:06:44.025440 27179 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 LEARNER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:44.028121 28529 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:06:44.030225 28518 raft_consensus.cc:979] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:44.031582 28576 raft_consensus.cc:1067] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:44.033545 26499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4429915312704856895d00171a6e6151"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:60744
I20240502 08:06:44.034152 26499 raft_consensus.cc:482] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:44.034543 26499 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:44.038280 26397 catalog_manager.cc:5484] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 7 to 8, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 8 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:44.058671 26499 raft_consensus.cc:504] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.061549 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:44.061853 26644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4429915312704856895d00171a6e6151" candidate_uuid: "8f9fe1f3763547388c4301b13a127aaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:44.062218 26571 raft_consensus.cc:3032] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:44.062453 26644 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:44.062559 26571 raft_consensus.cc:729] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:44.063410 26571 consensus_queue.cc:260] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.064519 26571 raft_consensus.cc:3037] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:44.068089 26650 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:44.072959 26568 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:06:44.073128 26499 leader_election.cc:288] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Requested vote from peers 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:44.073423 26644 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
I20240502 08:06:44.074856 26467 leader_election.cc:302] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 8f9fe1f3763547388c4301b13a127aaa; no voters:
I20240502 08:06:44.077607 28450 raft_consensus.cc:2781] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:44.073552 28530 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240502 08:06:44.111245 28529 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:44.120872 26571 raft_consensus.cc:2454] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9fe1f3763547388c4301b13a127aaa in term 4.
@ 0x7f96fda96fb7 gsignal at ??:0
I20240502 08:06:44.198333 28450 raft_consensus.cc:686] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [term 4 LEADER]: Becoming Leader. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:44.200094 28450 consensus_queue.cc:237] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.212133 28501 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.213222 26650 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.216677 26573 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.250460 27179 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.267352 26397 catalog_manager.cc:5484] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: config changed from index 3 to 4, 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } attrs { replace: true } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.284519 28530 raft_consensus.cc:979] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:44.285089 28586 raft_consensus.cc:1067] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:44.286849 26647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55500
I20240502 08:06:44.287359 26647 raft_consensus.cc:482] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:44.287680 26647 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:44.307744 26395 catalog_manager.cc:5484] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: term changed from 3 to 4, leader changed from 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2) to 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1). New cstate: current_term: 4 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:44.308492 26647 raft_consensus.cc:504] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.310530 26647 leader_election.cc:288] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2:40601)
I20240502 08:06:44.313028 26571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
I20240502 08:06:44.313591 26571 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Advancing to term 6
I20240502 08:06:44.316459 26496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2ed1efbf61440fbb8fbaf3ac77e227" candidate_uuid: "755f35bc66a841f382d1c4df6c6f6b64" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:44.317041 26496 raft_consensus.cc:3032] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Stepping down as leader of term 5
I20240502 08:06:44.317333 26496 raft_consensus.cc:729] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8f9fe1f3763547388c4301b13a127aaa, State: Running, Role: LEADER
I20240502 08:06:44.321211 26496 consensus_queue.cc:260] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
@ 0x7f96fda98921 abort at ??:0
I20240502 08:06:44.322544 26496 raft_consensus.cc:3037] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Advancing to term 6
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20240502 08:06:44.334952 26571 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
I20240502 08:06:44.340605 26616 leader_election.cc:302] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:44.341693 28534 raft_consensus.cc:2781] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 08:06:44.342231 28534 raft_consensus.cc:686] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [term 6 LEADER]: Becoming Leader. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:44.343116 28534 consensus_queue.cc:237] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:44.370510 26496 raft_consensus.cc:2454] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755f35bc66a841f382d1c4df6c6f6b64 in term 6.
@ 0x7f970088cda4 google::LogMessage::SendToLog() at ??:0
I20240502 08:06:44.411680 26492 consensus_queue.cc:237] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:44.449321 26644 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 755f35bc66a841f382d1c4df6c6f6b64 [term 8 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 08:06:44.451087 28522 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 08:06:44.454008 27179 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:06:44.456528 28501 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:44.466375 28450 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.478760 27179 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 5216c4a3a33f47ac88fa35704487f77e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.493532 26571 raft_consensus.cc:1261] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:06:44.501840 26492 raft_consensus.cc:1261] T 6f68e2a58df64f808deb0f0d3e63e43d P 8f9fe1f3763547388c4301b13a127aaa [term 8 FOLLOWER]: Refusing update from remote peer 97dc9560dbbe4d1aa44b2b2edda4080f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 08:06:44.505542 28530 consensus_queue.cc:1035] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:44.509074 28549 consensus_queue.cc:1035] T 6f68e2a58df64f808deb0f0d3e63e43d P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:06:44.509940 26385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a00601c8043478f8523c59e70966d50 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:06:44.517321 26397 catalog_manager.cc:5484] T 6a00601c8043478f8523c59e70966d50 P 8f9fe1f3763547388c4301b13a127aaa reported cstate change: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New cstate: current_term: 2 leader_uuid: "8f9fe1f3763547388c4301b13a127aaa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:44.529139 26631 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a00601c8043478f8523c59e70966d50 with delete_type TABLET_DATA_TOMBSTONED (TS 755f35bc66a841f382d1c4df6c6f6b64 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55494
I20240502 08:06:44.575603 26573 raft_consensus.cc:2932] T 6a00601c8043478f8523c59e70966d50 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } } }
I20240502 08:06:44.582373 26395 catalog_manager.cc:5484] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 reported cstate change: term changed from 5 to 6, leader changed from 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1) to 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3). New cstate: current_term: 6 leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: HEALTHY } } }
@ 0x7f970088d950 google::LogMessage::Flush() at ??:0
I20240502 08:06:44.622892 28600 tablet_replica.cc:330] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: stopping tablet replica
I20240502 08:06:44.624393 28600 raft_consensus.cc:2229] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:06:44.625253 28600 raft_consensus.cc:2258] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:06:44.631199 28600 ts_tablet_manager.cc:1875] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:06:44.650161 28600 ts_tablet_manager.cc:1888] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:06:44.650667 28600 log.cc:1196] T 6a00601c8043478f8523c59e70966d50 P 755f35bc66a841f382d1c4df6c6f6b64: Deleting WAL directory at /tmp/dist-test-taskCMs43N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714637090785390-25352-0/minicluster-data/ts-2-root/wals/6a00601c8043478f8523c59e70966d50
I20240502 08:06:44.657357 26386 catalog_manager.cc:4830] TS 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473): tablet 6a00601c8043478f8523c59e70966d50 (table test-workload [id=3ecee5abd3d343a3bf0a59433fdf8ae3]) successfully deleted
@ 0x7f9700892a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:06:44.746811 26573 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
@ 0x7f970121a3c7 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:06:44.781145 28450 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:06:44.837189 26571 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:44.838677 28534 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:44.846944 26644 raft_consensus.cc:1261] T 4429915312704856895d00171a6e6151 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Refusing update from remote peer 8f9fe1f3763547388c4301b13a127aaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
@ 0x7f97011f094f kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:06:44.872291 26496 raft_consensus.cc:1261] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 8f9fe1f3763547388c4301b13a127aaa [term 6 FOLLOWER]: Refusing update from remote peer 755f35bc66a841f382d1c4df6c6f6b64: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 08:06:44.873927 28414 consensus_queue.cc:1035] T 8e2ed1efbf61440fbb8fbaf3ac77e227 P 755f35bc66a841f382d1c4df6c6f6b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:06:44.903410 28450 consensus_queue.cc:1035] T 4429915312704856895d00171a6e6151 P 8f9fe1f3763547388c4301b13a127aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f97011ee0a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f970900e422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f9709014c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f97090aa4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f97090db3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f97090c11ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240502 08:06:45.365712 26615 consensus_peers.cc:485] T 33e1db2e4cc44834966a5a186294dff8 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
@ 0x7f97090c117a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
I20240502 08:06:45.403640 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
@ 0x7f97090c110a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f97090c10d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f97090c02fe std::__1::__function::__func<>::operator()() at ??:0
I20240502 08:06:45.530198 26573 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40c1d45639ea4980b7b376d3cd3cc1a3"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
mode: GRACEFUL
new_leader_uuid: "8f9fe1f3763547388c4301b13a127aaa"
from {username='slave'} at 127.0.0.1:41218
I20240502 08:06:45.530844 26573 raft_consensus.cc:595] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Received request to transfer leadership to TS 8f9fe1f3763547388c4301b13a127aaa
@ 0x7f970a466da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240502 08:06:45.544468 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "5216c4a3a33f47ac88fa35704487f77e"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:45.545152 26644 raft_consensus.cc:595] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Received request to transfer leadership to TS 5216c4a3a33f47ac88fa35704487f77e
I20240502 08:06:45.548148 26644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926"
dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
mode: GRACEFUL
new_leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:35282
I20240502 08:06:45.548730 26644 raft_consensus.cc:595] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Received request to transfer leadership to TS 97dc9560dbbe4d1aa44b2b2edda4080f
I20240502 08:06:45.557333 26496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6b6ee1a6eb54f00b15b4b47a4069d5b"
dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
mode: GRACEFUL
new_leader_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
from {username='slave'} at 127.0.0.1:55092
I20240502 08:06:45.564625 26496 raft_consensus.cc:595] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Received request to transfer leadership to TS 755f35bc66a841f382d1c4df6c6f6b64
I20240502 08:06:45.574167 28414 raft_consensus.cc:979] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:45.574801 28521 raft_consensus.cc:1067] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Signalling peer 97dc9560dbbe4d1aa44b2b2edda4080f to start an election
I20240502 08:06:45.576550 26573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926"
dest_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f"
from {username='slave'} at 127.0.0.1:51146
I20240502 08:06:45.577121 26573 raft_consensus.cc:482] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:45.577508 26573 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 4 FOLLOWER]: Advancing to term 5
@ 0x7f970a466bda std::__1::function<>::operator()() at ??:0
I20240502 08:06:45.597043 25352 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 08:06:45.598815 25352 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:06:45.599514 25352 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
@ 0x7f97011f2237 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 08:06:45.636157 26573 raft_consensus.cc:504] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
W20240502 08:06:45.645792 26615 consensus_peers.cc:485] T 2a6f8df55d8c4456ac8d84fdcb919e55 P 755f35bc66a841f382d1c4df6c6f6b64 -> Peer 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4:45711): Couldn't send request to peer 5216c4a3a33f47ac88fa35704487f77e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 08:06:45.649047 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:45.649739 26499 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Advancing to term 5
@ 0x7f97011f5692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f97011f564a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
I20240502 08:06:45.722577 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f7c38cbdaa43fa8b894d009abef926" candidate_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:45.723199 26650 raft_consensus.cc:3032] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 08:06:45.723469 26650 raft_consensus.cc:729] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:45.724232 26650 consensus_queue.cc:260] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
I20240502 08:06:45.725240 26650 raft_consensus.cc:3037] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Advancing to term 5
I20240502 08:06:45.686897 26499 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 8f9fe1f3763547388c4301b13a127aaa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:45.729074 26573 leader_election.cc:288] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:45.736548 26650 raft_consensus.cc:2454] T b4f7c38cbdaa43fa8b894d009abef926 P 755f35bc66a841f382d1c4df6c6f6b64 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dc9560dbbe4d1aa44b2b2edda4080f in term 5.
I20240502 08:06:45.738104 26542 leader_election.cc:302] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755f35bc66a841f382d1c4df6c6f6b64, 97dc9560dbbe4d1aa44b2b2edda4080f; no voters:
I20240502 08:06:45.739002 28549 raft_consensus.cc:2781] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 FOLLOWER]: Leader election won for term 5
I20240502 08:06:45.739547 28549 raft_consensus.cc:686] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 5 LEADER]: Becoming Leader. State: Replica: 97dc9560dbbe4d1aa44b2b2edda4080f, State: Running, Role: LEADER
I20240502 08:06:45.742254 28549 consensus_queue.cc:237] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } }
@ 0x7f97011f55da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f97011f55a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f97011f489e std::__1::__function::__func<>::operator()() at ??:0
I20240502 08:06:46.021386 28631 raft_consensus.cc:979] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64: : Instructing follower 5216c4a3a33f47ac88fa35704487f77e to start an election
I20240502 08:06:46.021921 28631 raft_consensus.cc:1067] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Signalling peer 5216c4a3a33f47ac88fa35704487f77e to start an election
I20240502 08:06:46.023692 27179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9"
dest_uuid: "5216c4a3a33f47ac88fa35704487f77e"
from {username='slave'} at 127.0.0.1:38552
I20240502 08:06:46.024209 27179 raft_consensus.cc:482] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:06:46.024565 27179 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 3 FOLLOWER]: Advancing to term 4
W20240502 08:06:46.024677 26542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 99451.2us
I20240502 08:06:46.032255 27179 raft_consensus.cc:504] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:46.037273 26397 catalog_manager.cc:5484] T b4f7c38cbdaa43fa8b894d009abef926 P 97dc9560dbbe4d1aa44b2b2edda4080f reported cstate change: term changed from 4 to 5, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 97dc9560dbbe4d1aa44b2b2edda4080f (127.24.194.2). New cstate: current_term: 5 leader_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dc9560dbbe4d1aa44b2b2edda4080f" member_type: VOTER last_known_addr { host: "127.24.194.2" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } }
I20240502 08:06:46.041028 26650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "755f35bc66a841f382d1c4df6c6f6b64"
I20240502 08:06:46.041605 26650 raft_consensus.cc:3032] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 08:06:46.041864 26650 raft_consensus.cc:729] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755f35bc66a841f382d1c4df6c6f6b64, State: Running, Role: LEADER
I20240502 08:06:46.042574 26650 consensus_queue.cc:260] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
I20240502 08:06:46.043635 26650 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:46.052656 26650 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 755f35bc66a841f382d1c4df6c6f6b64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
I20240502 08:06:46.053920 27145 leader_election.cc:302] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5216c4a3a33f47ac88fa35704487f77e, 755f35bc66a841f382d1c4df6c6f6b64; no voters:
I20240502 08:06:46.054771 28574 raft_consensus.cc:2781] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:06:46.039901 27179 leader_election.cc:288] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [CANDIDATE]: Term 4 election: Requested vote from peers 8f9fe1f3763547388c4301b13a127aaa (127.24.194.1:40577), 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3:40473)
I20240502 08:06:46.065280 26492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3c527a35284f68b946ea4bef6182c9" candidate_uuid: "5216c4a3a33f47ac88fa35704487f77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8f9fe1f3763547388c4301b13a127aaa"
I20240502 08:06:46.065907 26492 raft_consensus.cc:3037] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:06:46.074559 28501 raft_consensus.cc:979] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa: : Instructing follower 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:46.075156 28501 raft_consensus.cc:1067] T b6b6ee1a6eb54f00b15b4b47a4069d5b P 8f9fe1f3763547388c4301b13a127aaa [term 5 LEADER]: Signalling peer 755f35bc66a841f382d1c4df6c6f6b64 to start an election
I20240502 08:06:46.088956 26492 raft_consensus.cc:2454] T 6b3c527a35284f68b946ea4bef6182c9 P 8f9fe1f3763547388c4301b13a127aaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5216c4a3a33f47ac88fa35704487f77e in term 4.
@ 0x7f970a466da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240502 08:06:46.105280 28574 raft_consensus.cc:686] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [term 4 LEADER]: Becoming Leader. State: Replica: 5216c4a3a33f47ac88fa35704487f77e, State: Running, Role: LEADER
I20240502 08:06:46.117877 28574 consensus_queue.cc:237] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } }
W20240502 08:06:46.121631 26467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.194.3:40473, user_credentials={real_user=slave}} blocked reactor thread for 51720.8us
@ 0x7f970a466bda std::__1::function<>::operator()() at ??:0
I20240502 08:06:46.194386 26395 catalog_manager.cc:5484] T 6b3c527a35284f68b946ea4bef6182c9 P 5216c4a3a33f47ac88fa35704487f77e reported cstate change: term changed from 3 to 4, leader changed from 755f35bc66a841f382d1c4df6c6f6b64 (127.24.194.3) to 5216c4a3a33f47ac88fa35704487f77e (127.24.194.4). New cstate: current_term: 4 leader_uuid: "5216c4a3a33f47ac88fa35704487f77e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9fe1f3763547388c4301b13a127aaa" member_type: VOTER last_known_addr { host: "127.24.194.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755f35bc66a841f382d1c4df6c6f6b64" member_type: VOTER last_known_addr { host: "127.24.194.3" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5216c4a3a33f47ac88fa35704487f77e" member_type: VOTER last_known_addr { host: "127.24.194.4" port: 45711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:06:46.202755 28549 raft_consensus.cc:979] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f: : Instructing follower 8f9fe1f3763547388c4301b13a127aaa to start an election
I20240502 08:06:46.203275 28549 raft_consensus.cc:1067] T 40c1d45639ea4980b7b376d3cd3cc1a3 P 97dc9560dbbe4d1aa44b2b2edda4080f [term 3 LEADER]: Signalling peer 8f9fe1f3763547388c4301b13a127aaa to start an election
@ 0x7f97011d4937 kudu::Thread::SuperviseThread() at ??:0